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FOREWORD 


In  spite  of  significant  budgetary  restraints,  the  U.S.  Army  faces  the  challenges  of 
maintaining  combat  readiness  and  preparing  for  the  battlefield  of  the  21st  century.  In  fiscal  year 
1994,  Congress  appropriated  funding  for  the  Virtual  Brigade  Training  Program  (Department  of 
Defense,  October  1993).  Shortly  thereafter,  program  development  was  assumed  by  the  Force 
XXI  Training  Program  (FXXITP). 

Several  training  efforts  have  been  initiated  under  the  FXXITP.  One  of  those  efforts  is 
called  Combined  Arms  Operations  at  the  Brigade  Level,  Realistically  Achieved  Through 
Simulation  (COBRAS).  This  work  has  been  developed  by  the  U.S.  Army  Research  Institute  for 
the  Behavioral  and  Social  Sciences  and  the  U.S.  Army  Armor  Center  (Memorandum  of 
Agreement  entitled  “Force  XXI  Training  Program  [i.e..  Virtual  Brigade  Training  Program],”  June 
1994).  The  first  COBRAS  effort  —  COBRAS  I,  developed  two  types  of  exercises  designed  for 
the  staffs  of  inexperienced,  conventionally-equipped  brigades;  vignettes  for  segments  of  the  staff 
and  a  larger  Brigade  Staff  Exercise  (BSE)  focusing  on  the  primary  staff  members  plus  the  staff 
responsible  for  combat  support  (CS)  and  combat  service  support  (CSS).  Both  exercise  types 
offer  practice  and  feedback  opportunity  in  combat  fundamentals. 

The  second  COBRAS  effort  -  COBRAS  II,  the  subject  of  this  report  expands  on  the 
COBRAS  I  training  research  and  development,  implementing  lessons  learned  and  satisfying 
training  needs  (i.e.,  training  for  staff  personnel  linking  the  brigade  staff  and  additional  CS  and  CSS 
systems)  exposed  during  that  and  related,  concurrent  efforts. 

This  report  discusses  the  background  of  the  COBRAS  n  project  and  documents  the 
design  and  development  of  the  resulting  training  program.  Also  contained  are  lessons  learned  for 
future  development  efforts,  and  training  issues  that  will  likely  demand  attention  in  the  near  future. 
Force  XX3  policy  makers  and  training  developers  will  find  this  report  usefijl  in  the  course  of 
continuing  steady  progress  toward  Force  XXI  goals  in  training  and  readiness. 


ZITAM.  SIMUTIS 
Technical  Director 
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COMBINED  ARMS  OPERATIONS  AT  BRIGADE  LEVEL,  REALISTICALLY  ACHIEVED 
THROUGH  SIMULATION  II  (COBRAS  11):  REPORT  ON  DEVELOPMENT  AND  LESSONS 
LEARNED 


EXECUTIVE  SUMMARY 


Research  Requirement: 

In  1994,  the  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences,  in 
coordination  with  the  Force  XXI  Training  Program  and  the  U.S.  Army  Armor  Center,  launched  a 
research  and  development  effort  designed  to  help  brigade  staffs  become  proficient  in  the  combat 
fundamentals  that  will  be  required  on  the  digital  battlefield.  This  effort,  titled  Combined  Arms 
Operations  at  Brigade  Level  Realistically  Achieved  through  Simulation  (COBRAS),  is 
developing  and  evaluating  structured,  simulation-based  training  programs  and  strategies  to 
address  the  training  need. 

The  predecessor  project  (COBRAS  I)  developed  two  types  of  exercises:  a  Brigade  Staff 
Exercise  (BSE)  and  a  set  of  brigade  staff  vignettes  (Graves,  Campbell,  Deter,  &  Quinkert,  1997). 
The  COBRAS  n  project,  the  subject  of  this  report,  was  to  expand  on  that  work.  The  objectives 
of  the  COBRAS  II  project  were  to:  (a)  develop  a  BSE  that  provides  structured  training  for  a 
wider  audience,  (b)  develop  additional  vignettes  that  support  training  for  additional  audience,  (c) 
formatively  evaluate  the  newly-developed  training,  and  (d)  from  that  evaluation,  provide  lessons 
learned  and  insights  into  the  advancement  of  the  Force  XXI  training  strategy. 

Procedure: 

Working  from  the  COBRAS  I  products,  developers  identified  precisely  the  new  training 
audience  positions,  revised  the  scenario  to  provide  performance  opportunities  for  the  full 
audience,  and  revised  the  exercise  architecture  based  on  lessons  learned  in  the  initial  project.  This 
foundation  was  then  used  to  construct  the  training  support  package  (TSP)  materials  for  both  the 
BSE  and  the  vignettes. 

The  comprehensive  formative  evaluation  was  ongoing,  with  numerous  internal  and 
external  reviews  of  decisions  and  products.  Two  implementations  of  the  BSE  supported 
improvement  to  the  quality  and  functionality  of  the  training  model  and  TSPs.  Evaluation  of  the 
vignettes  depended  on  small  scale  pilot  tests,  supplemented  by  intensive  reviews  and  proofing 
within  the  project  staff 

Findings: 

The  design,  development,  and  evaluation  processes  described  above  combined  to  produce 
structured  exercises  and  TSPs  consistent  with  project  requirements.  Products  include  this  report 
and  other  documents  relating  to  the  development  methodologies  and  the  performance 
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requirements  (i.e.,  brigade  staff  task  lists)  produced  during  the  project,  in  addition  to  the  TSP 
materials  themselves. 

The  BSE  provides  an  opportunity  for  the  brigade  commander,  his  principal  staff,  and  the 
special  staff  who  serve  as  links  between  the  brigade  and  its  key  combat  support  and  combat 
service  support  systems  to  practice  their  roles  during  all  phases  of  a  three  mission  scenario.  The 
14  vignettes  were  developed  to  provide  structured  practice  opportunities  for  selected  segments  of 
the  brigade  staff  As  a  set,  the  vignettes  cover  general  staff  events  (e  g.,  plan  brigade  rear 
operations)  that  occur  during  the  three  mission  phases. 

Lessons  learned  during  the  project  and  insights  into  anticipated  training  needs  are 
presented  to  assist  training  developers,  including  Force  XXI  policy-makers,  as  they  continue  to 
advance  and  promote  the  Force  5DQ  Training  Program.  The  lessons  focus  on  the  development 
and  implementation  of  structured  simulation-based  training,  as  well  as  on  the  potential  value  of 
such  programs.  Insights  are  structured  around  ongoing  work  under  a  second  follow-on  project, 
COBRAS  III,  and  address  some  training  needs  that  may  surface  in  the  not  too  distant  fiiture. 

Utilization  of  Findings: 

This  report  presents  the  history  of  the  development  of  the  COBRAS  II  program  and  the 
lessons  learned  during  the  project's  duration.  As  continued  emphasis  is  placed  on  providing  low- 
resource,  cost-effective  training  for  U  S.  Army  staff  personnel,  this  report  can  lead  those  training 
development  efforts  into  the  selection  of  effective  design  and  implementation  strategies. 
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COMBINED  ARMS  OPERATIONS  AT  BRIGADE  LEVEL,  REALISTICALLY 
ACHIEVED  THROUGH  SIMULATION  H  (COBRAS  H): 

REPORT  ON  DEVELOPMENT  AND  LESSONS  LEARNED 
INTRODUCTION 

Military  forces  of  today  and  the  21st  century  are  at  a  decisive  inflection  point  in  how  they 
do  business.  It  is  characterized  by  the  transition  from  mostly  analog  battlefield  systems  to 
operations  with  the  new  digital  systems.  However,  before  the  transition  from  conventional  to 
digitized  systems  and  organizations  can  be  achieved,  Army  leaders  have  forecast  the  need  to 
strengthen  basic  combat  skills,  with  an  emphasis  on  command,  control,  and  communications  (C3). 
In  the  face  of  reduced  monetary  and  personnel  resources,  they  are  looking  to  simulation-based 
training  to  complement  institutional  training  and  live  field  training  exercises  in  preparing  soldiers 
for  the  demands  of  future  missions,  organizations,  and  weapon  systems. 

One  of  the  first  such  programs  was  the  Virtual  Training  Program  (VTP)*  developed  as  a 
research  and  development  (R&D)  product  by  the  Army  Research  Institute  for  the  Behavioral  and 
Social  Sciences  (ARI)  in  1993-1994.  The  VTP  provides  Army  National  Guard  and  active 
component  (AC)  units  with  high-quality,  time-compressed  structured  training  in  virtual  and 
constructive  environments  (Hoffman,  Graves,  Koger,  Flynn,  &  Sever,  1995).  Both  offensive  and 
defensive  exercises  were  developed  for  the  battalions,  battalion  staffs,  companies,  and  platoons. 

In  1995,  ARI  undertook  another  effort.  Combined  Arms  Operations  at  Brigade  Level, 
Realistically  Achieved  through  Simulation,  which  was  referred  to  as  COBRAS.  The  COBRAS 
project  developed  structured  simulation-based  training  exercises  for  the  staffs  of  conventional 
mounted  brigades.  The  exercises  included  a  Brigade  Staff  Exercise  (BSE)  and  vignettes  for 
subsets  of  the  brigade  staff  (Graves,  Campbell,  Deter,  &  Quinkert,  1997). 

The  follow-on  project,  COBRAS  H,  is  the  subject  of  this  report.  Expanding  on  the 
products  created  during  the  original  project^  the  COBRAS  H  effort  includes  an  expanded  BSE 
with  a  wider  training  audience  that,  in  effect,  replaces  the  initial  (COBRAS  I)  BSE,  and  additional 
brigade  staff  vignettes.^ 


*  The  program  was  originally  known  as  the  Reserve  Component  Virtual  Training  Program.  As  active  component 
(AC)  units  and  institutional  training  managers  became  interested  in  using  the  program,  the  “Reserve 
Component”  designation  was  dropped. 

^  In  this  report,  the  original  COBRAS  development  work  will  be  referred  to  as  COBRAS  I,  to  ensure  the  distinction 
between  the  two  efforts. 

^  The  COBRAS  n  project  also  enhanced  the  VTP  by  adding  opportunities  for  training  combat  support  (CS)  and 
combat  service  support  (CSS)  functions.  This  COBRAS  n  battalion-level  work  is  described  in  Hoffman’s  report. 
Combat  Support  and  Combat  Service  Support  Expansion  to  the  Virtual  Training  Program  SIMNET  Battalion 
Exercise:  History  and  Lessons  Learned  (1997).  The  products  of  that  effort  reside  with  the  VTP 
observer/controller  (0/C)  Team  of  the  16th  Cavalry  Regiment,  Fort  Knox. 
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Purpose  of  the  Report 


This  report  documents  the  development  process  and  lessons  learned  during  the  COBRAS 
n  project.  It  is  addressed  to  three  audiences:  the  sponsors  of  training  development  programs,  the 
developers  of  the  training  programs,  and  those  policy-makers  who  are  responsible  for  Army 
training  today  and  m  the  future.  In  addressing  those  audiences,  the  report  has  three  objectives: 

1.  to  detail  the  history  of  the  development  effort,  from  the  initial  design  activities  through 
the  production  and  delivery  of  the  final  training  support  package  (TSP)  materials; 

2.  to  describe  the  resulting  training  exercises  and  TSPs,  focusing  on  the  unique  aspects  of 
the  COBRAS  II  versus  the  COBRAS  I  program  designs;  and 

3.  to  present  lessons  learned  from  the  project,  provided  to  assist  Force  XXI  policy¬ 
makers  as  they  continue  to  advance  the  program,  and  to  aid  developers  of  other 
structured  simulation-based  training  programs. 

Organization  of  the  Report 

This  report  is  organized  in  seven  sections: 

•  Section  J:  The  COBRAS  II  Project.  Describes  the  early  stages  of  the  project  in  terms 
of  project  initiating  circumstances,  the  technical  objectives,  and  training  program 
design. 

•  Section  2:  COBRAS  II  Development  Plan  and  Evaluation  Plan.  Describes  the 
COBRAS  II  project's  development  methodology  and  presents  the  timeline  of  project 
events. 

•  Section  3:  Development  of  the  Brigade  Staff  Exercise.  Describes  the  development  of 
the  BSE  in  preparation  for  the  trial  implementation,  highlighting  how  and  why  the 
COBRAS  I  BSE  was  modified  from  its  original  version  to  achieve  the  objectives  of  the 
COBRAS  II  project. 

•  Section  4:  Trial  Implementation  of  the  Brigade  Staff  Exercise.  Describes  the  BSE 
trial  implementation  and  its  findings,  and  discusses  the  actions  taken  to  improve  the 
exercise  as  a  result  of  the  trial. 

•  Section  5:  Development  and  Pilot  Testing  of  the  Brigade  Staff  Vignettes.  Describes 
the  development  of  the  vignettes.  Modifications  to  the  COBRAS  I  development 
processes  are  highlighted.  Also  describes  the  vignette  pilot  implementations  and  their 
findings,  and  discusses  the  actions  taken  to  improve  the  vignettes  as  a  result  of  the 
pilots.  Included  is  a  discussion  of  how  certain  COBRAS  I  vignettes  were  re-examined 
and  modified  according  to  the  COBRAS  11  development  methodology. 

•  Section  6:  Lessons  Learnedfor  the  Development  of  Future  Training.  Summarizes 
lessons  learned  that  may  apply  to  future  training  development  projects.  Most  of  the 
lessons  relate  to  the  construction  and  content  of  TSPs  for  COBRAS-like  programs. 

•  Section  7:  Future  Directions.  Discusses  the  second  COBRAS  follow-on  effort, 
COBRAS  in,  and  project  notes  regarding  future  development  efforts. 
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Appendix  A  provides  definitions  of  the  acronyms  used  in  this  report.  Appendix  B, 
Foundations:  Overview  of  the  COBRAS  I  Program,  describes  the  original  project  in  order  to 
provide  a  brief  background  for  this  report.  It  includes  a  synopsis  of  the  COBRAS  I  program 
including  descriptions  of  the  exercises  and  TSPs.  The  discussion  also  summarizes  certain  aspects 
of  the  COBRAS  I  development  methodology  that  influenced  or  supported  continued  development 
in  the  COBRAS  11  effort.  Finally,  Appendix  C  provides  a  description  of  the  basic  methodology 
for  development  of  structured  simulation-based  training  and  a  description  of  the  specific  use  of 
the  methodology  for  development  of  structured,  small  group  exercises  (vignettes). 


3 


SECTION  1:  THE  COBRAS  H  PROJECT 


The  planned  scope  and  activities  of  the  COBRAS  II  project  were  outlined  by  ARI,  the 
United  States  Army  Armor  Center  (USAARMC),  and  the  Force  XXI  Training  Program  during 
the  early  stages  of  the  project.  During  this  time,  the  conceptual  foundations  of  the  program  were 
solidified  as  development  objectives  and  requirements.  This  section  describes  the  early  stages  of 
the  project,  in  terms  of  the  project  initiating  circumstances,  the  project's  technical  objectives,  and 
the  training  program  design. 


Project  Initiation 

In  early  1994,  with  the  support  of  the  Senate  Appropriations  Committee,  the  U.  S.  Army 
was  directed  to  “. . .  expand  the  existing  simulation  facilities  at  Fort  Knox,  develop  a  training 
strategy,  use  it  to  enhance  the  readiness  of  the  194th  Separate  Armor  Brigade,  and  evaluate  the 
effectiveness  of  these  simulations  and  this  new  strategy”  (Department  of  Defense,  October  1993). 
Congressionally-identified  funding  was  provided  to  the  ARI  -  Armored  Forces  Research  Unit  at 
Fort  Knox.  The  resulting  program,  known  as  the  Force  XXI  Training  Program,  was  seen  as  an 
Army-wide  program  for  integrating  virtual,  constructive,  and  live  simulation-based  training  into 
the  Force  XXI  plan  for  enhancing  combat  readiness.  The  effort,  conducted  by  ARI,  was  titled 
“Force  XXI  Training  Program  for  the  Conventional  Mounted  Brigade”^  and  began  in  January 
1995. 

The  COBRAS  11  project,  being  a  direct  extension  of  the  earlier  effort,  is  the  product  of  a 
continuance  of  the  congressionally  appropriated  funding.  Again,  ARI  defined  the  requirement  in  a 
statement  of  work  ([SOW]  ARI,  1995),  and  a  technical  response  (Human  Resources  Research 
Organization  [Hum^O],  1995)  was  prepared  to  detail  the  approach  to  meeting  the  requirement. 
That  technical  response  was  followed  by  preparation  of  the  research  plan  (HumRRO,  Hughes 
Training,  Inc.,  BDM  Federal,  Inc.,  &  PRC,  Inc.,  1996),  which  further  delineated  the  courses  of 
action  to  be  taken  to  fulfill  the  requirements  of  the  SOW.  It  also  provided  additional  detail 
regarding  the  proposed  design  of  the  COBRAS  II  program. 

The  SOW,  technical  response  to  the  SOW,  and  research  plan  Imd  down  the  first  detailed 
expectations  for  the  COBRAS  11  project.  These  expectations,  which  represent  the  purpose  and 
specifications  of  the  program,  are  described  below. 

Project  Purpose 

The  initial  COBRAS  I  program  was  designed  to  improve  the  decision-making  processes  of 
the  brigade  headquarters  through  the  use  of  simulation,  while  at  the  same  time  reducing  the 
amount  of  time  and  number  of  personnel  required  to  prepare  for  and  support  the  training.  At  a 
more  abstract  level,  however,  the  program  represented  a  test-bed  for  training  concepts  and 
training  technology.  That  is,  it  provided  an  initial  architecture  for  addressing  collective  training 
needs  for  a  conventional  brigade  staff  and  the  linkages  to  higher  and  subordinate  echelons. 


“  One  more  name  change  occurred  in  the  spring  of  1995,  when  the  program  title  Combined  Arms  Operations  at 
Brigade  Level,  Realistically  Achieved  Tlu-ough  Simulation,  and  its  acronym,  COBRAS,  came  into  common 
usage. 
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As  a  training  program,  the  COBRAS  I  exercises  provided  the  conditions  for  brigade  staff 
members  to  practice  decision-making,  communications,  synchronization,  and  integration 
processes  among  themselves  and  with  other  echelons.  However,  only  1 1  of  the  brigade  staff 
members  were  included  in  that  initial  effort.  The  research  requirement  for  the  COBRAS  II 
project  was  to  expand  the  exercise  to  include  five  more  members  of  the  staff,  representing 
communications,  chemical,  military  police  (MP),  army  aviation,  and  military  intelligence  (MI)  and 
electronic  warfare. 


COBRAS  n  Program  Design  Synopsis 

As  in  COBRAS  I,  the  COBRAS  II  exercises  were  to  be  designed  according  to  the 
principles  of  structured  training.  These  principles  include:  (a)  a  focus  on  performance  of  selected 
critical  tasks,  (b)  standardized  exercise  control  to  ensure  task  performance,  (c)  standardized 
feedback  to  correct  and  reinforce  task  performance,  and  (d)  exercise  support  by  means  of  a 
comprehensive  TSP. 

The  following  synopsis  provides  a  brief  overview  of  the  exercises  and  materials  to  be 
developed.  It  describes  the  types  of  exercises,  the  characteristics  of  the  program's  tactical 
scenario,  the  program's  instructional  design  characteristics,  and  the  characteristics  of  the  TSPs. 
The  program  was  to  be  a  direct  expansion  of  the  COBRAS  I  program;  that  original  program  is 
described  in  Appendix  B  to  provide  a  more  complete  background. 

Exercise  Types 

The  SOW  called  for  development  of  exercises  (i.e.,  a  BSE  and  vignettes)  of  the  same  type 
as  were  created  during  the  COBRAS  I  effort.  Like  those  exercises,  the  new  exercises  were  to  be 
differentiated  by  the  scope  of  their  focus. 

•  The  BSE  was  to  be  a  large  scale  training  exercise  that  would  immerse  the  brigade 
commander,  selected  members  of  his  staff,  and  key  leaders  with  links  to  various 
battlefield  operations  systems  (BOS)  in  a  multi-mission  scenario  covering  all  phases  of 
the  missions  (plan,  prepare,  and  execute,  including  consolidation  and  reorganization). 

•  The  vignettes  were  to  be  short  (less  than  one  training  day)  structured  exercises 
targeting  particular  groups  of  brigade  staff  members  on  specific  aspects  or  events 
within  the  context  of  the  plan,  prepare,  and  execute  phases  of  tactical  missions.  At 
least  two  vignettes  were  to  target  the  vertical  linkage  between  the  battalion  and 
brigade  echelons. 

Tactical  Scenario 

Both  types  of  exercises  were  to  be  based  on  a  single  scenario  that  would  represent  a 
variation  of  the  base  scenario  produced  during  the  initial  COBRAS  project.  In  that  scenario,  a 
conventional  mounted  brigade  is  opposed  by  a  Krasnovian  enemy  operating  according  to  the 
conventional  Soviet-style  threat  specifications  described  in  the  U.S.  Army  Training  and  Doctrine 
Command  (TRADOC)  Pamphlet  (PAM)  350-16  (Department  of  the  Army  [DA],  1994)  and  U.S. 
Army  Combined  Arms  Center  &  Fort  Leavenworth  PAM  350-1  (DA,  1993). 

The  scenario  includes  three  missions  (i.e.,  movement  to  contact  [MTC],  area  defense 
[AD],  and  deliberate  attack  [DATK]).  These  three  missions  are  incorporated  in  a  storyline 
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covering  a  continuous  war,  where  the  outcomes  of  one  mission  detemrune  the  location  and  general 
conditions  for  the  next  mission.  The  scenario  is  set  on  National  Training  Center  (NTC)  terrain 
and  covers  the  plan,  prepare,  and  execute  phases  of  each  mission,  with  a  focus  on  CS  and  CSS 
operations.  The  COBRAS  11  requirements  dictated  that  the  details  of  the  scenario  be  altered, 
however,  in  order  to  incorporate  events  (e.g.,  a  chemical  threat,  an  avdation  mission)  that  would 
stimulate  activities  for  the  new  training  audience  members. 

Instructional  Design 

The  instructional  design  of  the  program  was  to  be  consistent  with  the  design  of  the  initial 
COBRAS  program,  and  was  to  accommodate  both  the  expanded  training  audience  and  the 
lessons  learned  in  COBRAS  I.  The  exercises  were  to  be  exportable  and  implementable  without 
the  benefit  of  a  dedicated  observer/controller  (0/C)  team.  This  meant  that  all  participants, 
including  0/Cs,  would  have  to  come  from  within  the  training  brigade  or  its  division,  or  from  a 
sister  brigade.  It  also  meant  that  the  TSP  would  be  completely  self-contained,  requiring  no 
contractor  support  team  to  explain  how  to  implement  the  program.  The  training  objectives  and 
performance  observation  and  feedback  systems  were  to  be  based  on  the  design  of  the  COBRAS  I 
program. 

Trainin2  Support  Packages 

The  development  of  the  COBRAS  II  TSPs  was  to  start  with  the  materials  developed  in  the 
COBRAS  I  program.  The  model  for  the  BSE  included  a  single  TSP  that  would  be  flexible 
enough  to  be  used  under  different  implementation  configurations.  The  vignette  model  included  an 
individual  TSP  for  each  vignette,  linked  together  in  a  single  training  management  component. 

It  was  anticipated  that  new  ideas,  information,  and  instruction  would  emerge  during  BSE 
and  vignette  development,  because  of  the  expanded  training  audience  requirements,  and  would  be 
incorporated  into  the  TSPs.  Additionally,  job  aids,  guides,  and  task  lists  would  be  created,  as  they 
were  found  to  be  beneficial  or  necessary,  during  the  project. 

Development  Link  Between  Projects 

The  timing  of  the  two  projects  eased  the  transition  from  COBRAS  I  to  COBRAS  11  in  two 
ways.  First,  the  trial  implementation  of  the  COBRAS  I  BSE  did  not  occur  until  after  the 
conclusion  of  the  COBRAS  I  project.  Second,  the  development  of  the  COBRAS  I  vignettes  for 
constructive  simulation  environments  was  not  completed  during  the  timeframe  of  the  first  project, 
but  was  assimilated  into  the  development  of  the  COBRAS  II  vignettes. 

The  final  trial  of  the  COBRAS  I  BSE  was  conducted  with  members  of  the  2"*'  Brigade,  1“ 
Infantry  Division  (ID)  (Mechanized),  Fort  Riley  in  August  1996.  In  fact,  this  was  the  only 
implementation  of  the  COBRAS  I  BSE  with  an  intact  brigade  staff.  The  other  implementation 
was  an  early  pilot  test  conducted  with  members  of  the  16th  Cavalry  at  Fort  Knox,  who  were 
generally  battalion-level  personnel  roleplaying  a  brigade  staff 

At  the  time  of  the  COBRAS  I  final  trial,  the  TSP  was  complete  and  had  undergone 
thorough  review,  in  terms  of  tactics  and  instructional  quality  issues,  by  the  brigade-level  subject 
matter  expert  (SME)  personnel  on  the  COBRAS  team  as  well  as  external  reviewers.  Because  the 
project  had  already  been  concluded,  and  because  the  COBRAS  II  BSE  was  to  replace  the 
COBRAS  I  BSE,  the  strategy  for  utilizing  trial  results  was  to  incorporate  it  in  designing  and 
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developing  the  COBRAS  n  exercise  and  TSP,  rather  than  using  the  information  to  revise  the 
COBRAS  I  products  * 

The  COBRAS  I  project's  effort  to  develop  two  vignettes  to  be  conducted  in  constructive 
simulation  environments  was  delayed,  in  part  due  to  difficulties  in  gaining  access  to  the  simulation 
systems  for  development  and  tryout,  and  also  because  of  the  difficulty  of  the  task  and  the 
unexpected  time  required  to  develop  short  duration  exercises  that  use  constructive  simulation. 
These  factors  extended  the  development  time  frame.  Early  in  the  COBRAS  n  project,  the 
decision  was  made  to  complete  development  of  the  two  COBRAS  I  simulation-based  vignettes  at 
the  same  time  as  the  development  of  similar  COBRAS  II  vignettes.  It  was  expected  that  lessons 
learned  during  the  development  of  the  COBRAS  II  BSE  and  vignettes  would  contribute  to 
improving  both  sets  of  simulation-based  vignettes. 

Summary 

The  COBRAS  11  design  and  development  process  was  a  carefully  considered  extension  of 
COBRAS  I  work.  From  its  inception,  the  COBRAS  U  BSE  was  intended  to  be  the  surviving 
exercise  package,  with  the  COBRAS  I  BSE  relegated  to  archives  and  of  historical  interest  only. 
However,  the  COBRAS  11  vignettes  would  be  added  to  the  previously  constructed  set.  In  the 
following  section,  the  description  of  the  COBRAS  II  development  plan  demonstrates  clearly  the 
similarities  between  the  projects,  in  terms  of  the  development  methodology  employed  and  the 
evaluation  plan. 


*  Although  areas  for  improvement  in  the  COBRAS  I  BSE  materials  were  identified,  none  precluded  the  use  of  tlie 
BSE  at  an  appropriate  point  in  a  brigade’s  training  strategy. 
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SECTION  2:  COBRAS  H  DEVELOPMENT  AND  EVALUATION  PLAN 


Like  its  predecessor,  COBRAS  II  employed  the  methodology  for  the  development  of 
structured  simulation-based  training  (C.  H.  Campbell,  Campbell,  Sanders,  Flynn,  &  Myers,  1995), 
which  was  an  outcome  of  the  development  of  the  VTP.  COBRAS  II,  however,  was  able  to 
benefit  from  lessons  learned  during  the  application  of  the  methodology  in  the  first  COBRAS 
project,  as  documented  in  a  revised  methodology  (C.  H.  Campbell  &  Deter,  1997; 

C.  H.  CampbeU,  Deter,  &  Quinkert,  1997). 

Consistent  with  both  versions  of  the  methodology  and  the  project  SOW,  the  conduct  of  a 
formative  evaluation  was  a  vital  aspect  of  the  COBRAS  II  effort.  The  evaluation  strategy 
required  iterative  implementations  and  reviews  of  the  training  throughout  the  course  of  its 
development.  The  findings  from  the  evaluation  were  key  to  improving  the  program  in  terms  of  its 
task  focus  and  implementation  approach. 

This  section  of  the  report  presents  the  development  method  and  timeline  of  the  COBRAS 
n  project.  The  three  major  topics  include; 

•  Development  Methodology.  A  description  of  the  structured  simulation-based  training 
development  methodology. 

•  Formative  Evaluation  Approach'.  Describes  the  evaluation  approach  and  provides  a 
brief  introduction  to  the  evaluation  procedures  performed  during  the  COBRAS  II 
project. 

•  Development  Process:  Describes  the  major  project  events  and  phases. 

Development  Methodology 

The  revised  development  methodology  (C.  H.  Campbell,  Deter,  &  Quinkert,  1997)  was  to 
drive  the  COBRAS  II  project's  development  of  the  BSE  and  vignettes.  (For  readers  unfamiliar 
with  the  methodology,  a  description  is  presented  in  Appendix  C.)  Because  the  COBRAS  II  BSE 
represented  an  expansion  of  the  COBRAS  I  exercise,  a  modified  application  of  the  methodology 
was  employed  to  account  for  the  fact  that  many  aspects  of  the  exercise  (e.g.,  scenario, 
performance  job  aids)  had  been  previously  constructed  and  needed  only  modification.  For  the 
vignettes,  however,  the  methodology  was  to  be  followed  in  full;  that  is,  each  of  the  vignettes  was 
to  be  developed  from  an  initial  concept  and  would  therefore  require  a  full  implementation  of  the 
methodology. 


Formative  Evaluation  Approach 

An  integral  and  critical  aspect  of  development  was  the  formative  evaluation.  In  the 
context  of  structured  training  development,  “formative  evaluation”  means  an  ongoing  cycle  of 
examine-evaluate-refine.  Formative  evaluation  occurs  not  after  a  project’s  completion,  but  during 
the  development  and  initial  implementations  of  the  program. 

The  evaluation  method  for  the  COBRAS  II  program  was  based  on  the  formative 
evaluation  process  outlined  by  Herman,  Morris,  and  Fitz-Gibbon  (1987),  modified  to  be 
consistent  with  the  method  used  in  the  creation  of  the  VTP  (Hoffman  et  al.,  1995).  The  examine- 


8 


evaluate-refine  cycle  was  executed  throughout  every  phase  of  design  and  production  to  ensure 
that  each  component  of  the  BSE  and  vignettes  was  functional,  doctrinally  correct,  and  consistent 
with  other  components. 

The  COBRAS  n  formative  evaluation  began  informally  with  assessments  of  how  to 
implement  the  initial  design  decisions  consistent  with  and  supportive  of  the  project's  objectives. 

For  example,  inclusion  of  additional  training  audience  members  would  not  be  pemutted  to  reduce 
the  amount  of  training  for  the  original  training  audience.  As  new  concepts  and  training  materials 
were  developed,  further  assessments  were  made  regarding  how  these  products  would  support  the 
training  objectives  and  the  fiirther  development  of  the  program.  These  early  evaluation  activities 
were  primarily  internal  to  the  development  team,  with  briefinp  and  guidance  from  the  Force  XXI 
Training  Program  and  the  USAARMC  at  critical  decision  points. 

As  development  efforts  proceeded,  evaluation  acti\dties  became  more  formal  and 
comprehensive  in  nature,  and  increasingly  significant  external  support  was  required.  These 
included  two  types  of  evaluation  activity:  pilot  exercises  and  trial  implementations.  The  conduct 
and  results  of  the  evaluation  activities  are  presented  in  Sections  4  and  5,  and  are  summarized 
below. 

For  the  BSE,  pilot  exercises  were  conducted  with  program  developers  assisted  by 
developers  of  other  structured  training  programs,  simulation  site  staff,  and  military  personnel  from 
the  USAARMC.  Vignette  pilot  tests  (conducted  for  only  five  of  the  vignettes)  involved 
COBRAS  developers,  lb*  Cavalry  Regiment  (Fort  Knox)  0/Cs  from  the  VTP,  and  military 
personnel  who  were  representative  of  the  user  population.  In  both  cases,  the  focus  was  on 
assessment  of  the  correctness  and  usability  of  selected  components  of  the  exercises. 

Trial  implementations  usually  involve  users  representative  of  the  intended  training 
participants,  and  are  intended  to  allow  evaluation  of  all  components  of  the  TSP.  A  trial 
implementation  provides  the  forum  for  obtaining  comprehensive  user  input  reprding  efficacy  of 
design  decisions,  material  functionality,  user  acceptability,  and  perceived  training  benefit.  For  the 
BSE,  a  trial  implementation  of  one  mission  was  conducted.  None  of  the  vignettes  had  a  full  trial 
implementation;  instead,  intensive  project  staff  reviews  were  conducted. 

The  methods  employed  to  collect  information  changed  between  the  early  informal 
evaluation  processes  and  the  pilots  and  trials.  During  early  stages  of  development,  the  activities 
were  relatively  unstructured,  and  very  little  systematic  data  collection  and  analysis,  in  a  traditional 
sense,  was  performed.  Developers  focused  their  attention  on  specific  processes  that  they  needed 
to  assess.  For  example,  in  early  simulation-supported  tryouts  of  the  scenario  settings,  the  scenario 
contained  initial  conceptions  of  events  that  were  to  occur  to  drive  the  activities  of  the  newly 
added  target  training  audience  personnel.  The  evaluations  were  aimed  at  determining  if  the 
simulation  representation  of  the  scenario  would  function  as  intended,  providing  cues  and 
conditions  as  designed.  A  scenario  might  be  tested,  revised,  and  tested  again  in  the  same  day, 
consequently,  the  focus  was  on  obtaining  and  applying  information  rapidly. 

During  the  latter  pilot  implementation,  however,  the  data  collection  was  more  systematic 
and  specific.  Expected  outcomes  were  targeted  before  each  pilot  or  trial,  in  the  form  of  specific 
questions,  and  evaluation  instruments  were  designed  to  facilitate  the  collection  of  information  to 
address  the  issues.  Generally,  pilot  results  included  corrections,  clarifications,  resolution  of 
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inconsistencies,  amplifications  or  elaboration,  deletions,  and  development  of  new  materials  and 
procedures  as  solutions  to  specific  design  problems. 

The  trial  implementation  was  larger  in  scope  than  the  pilots,  and  consequently  involved 
more  extensive  formative  evaluation  procedures.  The  formative  evaluation  instruments  for  the 
trial  were  designed  to  support  both  observational  and  querying  methods  of  data  collection.  The 
instruments  generally  included  structured  interview  guides,  evaluator  guides,  questionnaires,  and 
group  discussion  guides.* 

Each  evaluation  exercise  was  designed  for  timely  documentation  of  results  so  that 
information  could  be  disseminated  quickly  among  the  training  developers.  Involvement  of  the 
training  developers  in  the  observation  process  facilitated  the  extraction  of  findings  and  their 
translation  into  steps  for  modification.  The  observations  and  data  from  each  exercise  were 
organized  and  analyzed  rapidly.  The  findings  were  then  distributed  immediately  to  the  training 
developers  and  dialog  sessions  were  organized,  as  necessary,  to  clarify  and  prioritize  revision 
actions. 

Development  Process 

As  described  above,  the  project  followed  the  methodology  for  development  of  structured 
simulation-based  training.  The  methodology  is  not  a  strictly  linear  procedure,  but  rather 
anticipates  considerable  modifications  to  design  decisions  as  further  development  continues. 

The  first  phase  of  the  project  stretched  through  the  first  eight  months  of  the  project. 
During  this  phase,  various  design  decisions  were  delineated  pertaining  to  the  tr^ning  audience 
definition,  the  training  objectives,  the  scenario  specifications,  and  the  TSP  architecture.  Part  of 
the  reason  for  the  seemingly  inordinate  time  requirement  was  that  the  COBRAS  I  effort  itself  was 
still  underway  when  COBRAS  II  work  began.  Thus  information  affecting  COBRAS  II  decisions 
was  still  being  generated  by  the  COBRAS  I  project. 

A  critical  event  was  the  conduct  of  the  trial  of  the  COBRAS  I  BSE  at  Fort  Riley,  KS,  in 
August  1996.  During  the  trial,  developers  assessed  the  implementation  with  an  eye  towards  how 
the  exercise  should  be  modified  to  represent  a  functional  COBRAS  II  BSE. 

The  actual  development  of  the  COBRAS  scenario,  delineation  of  the  training  objectives, 
and  construction  of  the  TSP  for  the  BSE  and  vignettes  began  well  before  the  COBRAS  I  BSE 
was  implemented  at  Fort  Riley.  The  design  processes  of  performance  analysis  and  scenario 
construction  were  overlaid  with  TSP  construction  in  order  to  meet  a  trial  implementation  set  to 
occur  only  four  months  after  the  Fort  Riley  trial.  The  initial  development  of  the  BSE,  along  with 
the  development  of  the  scenario  and  task  analyses  is  described  in  Section  3. 

The  trial  implementation  of  the  BSE,  described  in  Section  4,  occurred  as  planned  at  Fort 
Lewis,  WA,  in  December  1996.  Participants  included  members  of  the  3"*  Brigade,  2"*  ED  and  the 
181“  Brigade  Reserve  Training  Detachment.  The  trial  enabled  developers  to  try  out  and  obtain 
information  on  many  aspects  of  the  exercise,  including  the  training  objectives  and  tasks,  but 
excluding  the  scenarios  of  the  MTC  and  AD  missions.  After  the  trial,  intense  internal  reviews  of 


®  Questionnaires  were  review  ed  by  ARI  prior  to  the  trial  to  ensure  rights  of  the  respondents  would  be  protected. 
The  questionnaires  were  assigned  ARI  Personnel  Test  Number  60-02. 
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the  TSP  were  conducted,  and  data  were  analyzed  and  applied  to  improve  the  exercise.  Feedback 
regarding  the  DATK  (the  mission  executed  during  the  trial)  scenario,  was  extrapolated  to  assist  in 
the  refinement  of  the  other  two  missions  as  well  as  the  DATK. 

The  BSE  received  another  significant  evaluation  with  the  same  brigade  along  with  0/C 
personnel  from  I  Corps  after  the  project  was  considered  finished  (in  May  1997).  Section  7  of  this 
report  describes  this  implementation  and  its  application  to  future  generation  COBRAS  programs. 

Significant  work  on  the  vignettes  began  with  identification  of  vignette  concepts,  or  topics 
of  focus.  The  more  resource  demanding  vignette  efforts  were  initiated  after  the  BSE  trial.  It  was 
during  this  period  that  the  concepts  and  scopes  of  the  individual  vignettes  were  refined  and  the 
associated  TSPs  constructed.  This  initial  vignette  development  work  is  described  in  Section  5  of 
this  report. 

Pilots  of  the  vignettes  involved  military  SMEs  in  armor,  field  artillery,  chemical 
operations,  CSS,  and  intelligence.  The  number  of  available  U.S.  Army  participants  and  their 
expertise  supported  pilot  implementation  of  only  the  five  vignettes  that  focused  on  those  areas. 
Again,  however,  the  results  were  extrapolated  and  applied  as  appropriate  to  refine  all  the 
COBRAS  II  vignettes. 

The  internal  reviews  and  refinement  of  the  vignettes,  after  the  pilots,  was  a  particularly 
important  aspect  of  their  development.  These  reviews  examined  the  extent  to  which  each  vignette 
facilitated  training  for  its  audience.  At  the  conclusion  of  the  reviews,  each  vignette  had  been 
substantively  revised,  relying  heavily  on  the  feedback  received  during  the  external  pilots.  The  four 
vignettes  designed  for  constructive  simulation,  including  the  two  that  were  incomplete  from  the 
COBRAS  I  project,  were  the  last  vignettes  to  be  completed. 

Throughout  the  project.  Force  XXI  Training  Program  and  USAARMC  were  kept 
informed  on  the  progress  and  obstacles  by  means  of  a  series  of  in-process  review  (IPRs). 

Summary 

Within  the  development  and  evaluation  plan  described  here,  there  were  various 
considerations  and  decisions  that  shaped  the  final  products.  The  next  three  sections  of  the  report 
describe  the  design  and  development  events,  so  that  future  developers  may  have  a  more  complete 
picture  of  the  processes  and  issues  they  may  confront.  Sections  3  and  4  describe  the  production 
of  the  BSE  and  Section  5  describes  the  production  of  the  vignettes.  Following  the  development- 
oriented  sections,  the  report  provides  lessons  learned  and  a  description  of  the  needs  that  remain  to 
be  addressed  by  future  R&D. 
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SECTION  3:  DEVELOPMENT  OF  THE  BRIGADE  STAFF  EXERCISE 

Because  the  developmental  process  for  COBRAS  n  is  very  similar  to  the  COBRAS  I 
process  (Graves  et  al.,  1997),  this  report  highlights  the  aspects  of  development  and  the  exercise 
itself  that  were  unique  to  the  COBRAS  11  effort.  It  identifies  those  aspects  of  the  BSE  that  were 
changed  during  the  project,  how  they  were  changed,  and  the  outcomes  of  those  changes  (i.e.,  new 
or  revised  TSP  components). 

This  section  deals  with  BSE  development  in  preparation  for  the  trial  implementation  at 
Fort  Lewis,  WA  in  December  1996.  The  topics  covered  include: 

•  Designating  the  Primary  Training  Audience'.  Describes  how  SOW  identification  of 
additional  brigade  functional  areas  for  the  exercise  evolved  into  designation  of  the 
specific  new  audience  members. 

•  Designing  the  Tactical  Scenario:  Explains  how  the  COBRAS  I  scenario  was  altered 
to  include  scenario  events  that  would  cue  the  involvement  of  the  new  training  audience 
members. 

•  Designating  the  Training  Objectives:  Describes  the  task  identification  and  selection 
process  and  its  products. 

•  Designing  the  Exercise  Architecture  :  Presents  the  architecture  of  the  COBRAS  II 
BSE  as  it  was  prepared  for  the  trial,  and  how  it  differed  from  that  of  the  original  BSE. 

•  Developing  the  Training  Support  Package  :  Provides  a  synopsis  of  the  BSE  TSP  as  it 
was  prepared  for  the  trial.  Special  focus  is  placed  on  describing  how  the  COBRAS  11 
TSP  differed  from  the  COBRAS  I  version. 

Designating  the  Primary  Training  Audience 

The  first  and  most  consequential  decision  in  designing  the  BSE  was  identifying  new 
training  audience  members.  The  SOW  identified  five  functions  that  were  to  be  represented  by  the 
new  training  audience;  signal,  chemical,  MP,  Army  aviation,  and  intelligence  and  electronic 
warfare.  Specific  training  audience  positions  to  represent  the  functions  were  identified  by 
reference  to  three  criteria: 

•  First,  the  new  audience  members  had  to  be  active  participants  in  the  staff  process, 
which  was,  from  the  beginning,  the  focus  of  the  BSE.  The  members  might  be 
peripheral  participants,  or  involved  only  in  selected  activities,  but  they  had  to  be 
involved  in  the  direct  interaction  that  takes  place  among  the  brigade  staff. 

•  Second,  the  new  members  had  to  be  active  throughout  the  majority  of  the  exercise. 
This  would  ensure  that  the  training  benefit  would  be  worth  the  time  required  for 
preparation  and  participation  in  the  exercise. 

•  Third,  each  new  audience  member's  role  in  the  exercise  had  to  be  consistent  with 
current  and  emerging  (to  the  extent  possible)  doctrine. 

The  research  into  the  various  functional  areas  included  thorough  job  analysis  of  the 
principal  participants  in  each  function,  primarily  through  the  relevant  field  manual  and  Army 
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training  and  evaluation  program-mission  training  plan  publications.  As  the  selection  process 
proceeded,  developers  consulted  and  maintained  contact  with  representatives  of  the  proponent 
agency  schools  (e.g.,  the  U.S.  Army  Intelligence  and  Electronic  Warfare  School  at  Fort 
Huachuca).  These  relationships  ensured  that  the  most  current  doctrinal  information  was 
considered,  thereby  facilitating  the  decision-making  process.  Eventually,  the  following  individuals 
were  added  as  training  audience  members: 

•  the  signal  officer, 

•  the  MP  platoon  leader, 

•  the  assistant  operations  and  training  officer  (S3)/chemical  officer  (CHEMO), 

•  the  MI  company  commander,  and 

•  the  Army  aviation  liaison  officer  (AVN  LNO). 

Representatives  of  the  Force  XXI  Training  Program  and  the  USAARMC  concurred  with 
the  selection  of  the  new  audience  members  as  well  as  the  rationale  for  their  selection.  This  was  an 
important  step  in  ensuring  that  the  expectations  of  the  proponent  agencies  and  program 
stakeholders  would  be  met.  The  expanded  training  audience  is  shown  in  Figure  1. 

Designing  the  Tactical  Scenario 

The  COBRAS  II  scenario  was  not  developed  from  scratch,  but  represented  a  revision,  or 
more  accurately,  an  expansion  of  the  COBRAS  I  scenario.  In  the  early  stages  of  scenario 
redesign,  efforts  were  driven  by  two  distinct  purposes: 

1 .  to  include  events  for  the  new  training  audience  members,  and 

2.  to  incorporate  improvements  based  on  findings  obtained  during  the  COBRAS  I  trial 
implementation. 

The  first  purpose  (above)  required  an  analysis  of  how  well  the  existing  storyline  and  series 
of  events  would  involve  the  new  training  audience  members,  drawing  them  into  the  staff  process. 
Scenario  events  that  simply  gave  them  something  to  do  were  not  sufficient.  This  analysis 
depended  on  a  general,  comprehensive  job  and  task  analysis  for  each  of  the  five  positions,  derived 
from  Army  documentation  and  from  discussions  with  SMEs.  Further  progress  was  made  during 
the  COBRAS  I  trial,  during  which  developers  monitored  the  implementation  while  envisioning 
ways  to  tie  in  the  performance  of  the  new  audience  members. 

The  integration  of  the  S3/CHEMO  was  judged  to  require  the  most  extensive  changes  to 
the  scenario.  The  major  change  would  be  including  a  threat  of  opposing  force  (OPFOR)  chemical 
operations.  It  seemed  likely  that,  during  further  development,  the  unchecked  use  of  threat 
chemical  weapons  could  alter  the  scenario  timeline  and  become  the  dominant  focus  of  the  brigade 
staffs  efforts.  Throughout  the  development  process,  developers  watched  carefully  for  unintended 
effects  of  introducing  chemical  events  into  the  scenario. 
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Figure  1.  COBRAS  I  and  COBRAS  11  primary  training  audience  links  to  brigade  assets. 


On  the  other  hand,  the  COBRAS  I  scenario  appeared  to  offer  sufficient  opportunities  and 
cues  for  the  activities  of  the  signal  officer  and  the  AVN  LNO.  These  two  positions  were  expected 
to  be  able  to  maintain  their  activities  to  an  adequate  extent  within  the  existing  maneuver  and  close 
air  support  aspects  of  the  scenario. 

Integration  of  the  MP  platoon  leader  would  require  the  addition  of  tactical  activities  in  the 
brigade's  rear.  These  activities  would  keep  the  MP  platoon  leader  involved  in  the  staff  process 
without  altering  significantly  the  overall  activity  of  the  brigade.  Similarly,  activities  and  cues  to 
ensure  integration  of  the  MI  company  commander  could  be  accomplished  without  major  changes 
in  the  scenario.  The  MI  company  commander  would  be  provided  the  typical  company-level  assets 
and  would  interact  with  the  staff  in  matters  of  intelligence  reports.  His/her  presence  would  alter 
only  the  brigade's  task  organization,  while  leaving  intact  the  remainder  of  the  scenario. 

Incorporating  these  new  events  and  opportunities  for  performance  was  the  major  push  in 
the  scenario  design  effort.  Equally  important,  however,  was  the  need  to  preserve  the  structure  of 
the  exercise  in  terms  of  other  design  considerations,  including  the  contiguous-mission  design,  time 
requirements,  and  provision  of  training  opportunities  for  the  original  training  audience  members. 
The  contiguous  nature  was  required  to  cue  CS  and  CSS  activities,  and  the  time  required  for  a 
mission  had  to  be  bounded  to  maintain  the  exercise's  appeal  to  brigades  as  something  they  could 
accomplish  within  their  limited  time  and  simulation  resources. 
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The  second  purpose  of  the  scenario  design  effort,  to  incorporate  the  necessary 
modifications  and  improvements  identified  during  the  COBRAS  I  trial,  was  addressed 
immediately  following  the  trial,  and  continued  well  into  the  scenario  design  process.  Observations 
and  participant  feedback  about  how  the  scenario  could  be  improved  were  compiled  and  discussed 
in  order  to  identify  suggestions  that  were  valid  and  consistent  with  the  intent  for  the  COBRAS  n 
scenario.  The  major  changes  are  summarized  in  Figure  2. 

Work  to  integrate  new  events  and  other  improvements  into  the  scenario  began  with  the 
revision  of  tactical  products,  starting  with  the  corps  and  division  operations  order  (OPORD) 
materials.  Other  aspects  of  the  scenario,  such  as  products  that  contained  ancillai7  information 
(e.g.,  personnel  status,  levels  of  supply  and  maintenance,  intelligence  summary  information, 
meteorological  and  terrain  information)  that  would  flow  down  from  division  and  corps,  were  also 
altered  to  provide  performance  cues  as  necessary.  Together,  these  products  were  the  structuring 
mechanism  of  the  scenario,  as  they  set  the  situation  and  conditions  under  which  a  brigade  would 
develop  its  plan. 

The  initial  testing  of  the  scenario  occurred  in  map  exercises  and  staff  roleplay  enactments. 
Map  exercises  were  conducted  to  ensure  that  the  scenario  was  tactically  sound  and  that  the 
actions  required  of  the  brigade  were  consistent  with  current  doctrine,  as  well  as  internally 
consistent.  The  scenario  was  also  reviewed  to  ensure  it  would  not  lead  the  training  brigade 
outside  of  the  context  of  the  contiguous  mission  concept.  The  most  important  focus,  however, 
was  on  verifying  that  the  scenario  would  contain  the  battle  events  that  would  cue  activity  within 
the  brigade  staff,  including  CSS  functions. 

The  roleplay  enactments,  which  were  a  critical  part  of  the  task  identification  effort 
(described  below),  provided  opportunities  to  examine  the  extent  to  which  the  scenario's  events 
would  provide  performance  opportunities  for  the  new  training  audience  personnel.  At  the  same 
time,  attention  was  also  paid  to  maintaining  the  performance  requirements  of  the  original  training 
audience.  Notably,  there  were  few  instances  in  which  the  revised  scenario  altered  fundamentally 
the  performance  requirements  of  these  positions. 

Following  the  initial  map  exercises  and  roleplay  enactment  time  frame,  and  when  the 
tactical  products  were  judged  as  being  sufficiently  complete,  simulation-based  tryouts  were 
conducted,  much  like  those  conducted  during  the  COBRAS  I  project.  The  tiyouts  required  the 
development  of  BBS  electronic  files  which  contained  the  details  of  the  task  organization, 
readiness  levels,  and  operational  graphics.  Their  purpose  was  to  ensure  that  the  scenario  would 
unfold  as  designed,  producing  the  desired  events  and  cues. 
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Scenario  Revisions  Based  on  COBRAS  I  Trial 

•  Reduced  the  brigade’s  initial  readiness  levels  in  all  missions  to  force  more  CSS  activity. 

•  Modified  OPFOR  routes  to  refine  the  timing  and  locations  of  confrontations  with  the 
brigade. 

•  Moved  the  brigade’s  air  avenue  of  approach  to  make  close  air  support  more  realistic. 

•  Extended  engineer  obstacle  zone  to  influence  the  outcome  of  the  fight  in  the  area  defense 
mission. 

•  Modified  field  artillery  Family  of  Scatterable  Mines  (FASCAM)  authorities  to  better  define 
conditions  of  release  and  deployment. 

•  Clarified  the  limit  of  advance  in  terms  of  times,  locations,  and  size  of  forces. 

•  Included  a  tactical  combat  force  in  55  Division  to  match  doctrinal  specifications. 

•  Clarified  contact  points. 

•  Provided  more  information  on  activities  of  adjacent  units  by  means  of  scripted  messages 
fi-om  the  Exercise  Control  (EXCON). 

•  Revised  the  scripted  message  to  cause  intelligence  summaries  to  be  sent  at  regular  intervals. 

Figure  2.  Scenario  revisions  based  on  COBRAS  I  Brigade  Staff  Exercise  trial  results. 

Mission  by  mission,  the  scenario  was  evaluated  for  its  capacity  to  logically  and  realistically 
contain  the  events  that  would  cue  the  intended  performance  requirements.  The  resulting  scenario 
and  tactical  products  were  judged  to  be  appropriate  to  support  continuing  development. 

COBRAS  n  Scenario  Storyline 

The  COBRAS  11  scenario,  with  all  the  changes  made  during  that  effort,  formed  the  basis 
of  the  COBRAS  11  BSE.  It  also  provided  the  framework  for  how  the  higher  echelon,  adjacent 
and  supporting  units,  subordinate  units,  and  OPFOR  were  to  operate  in  the  vignettes.  Although 
individual  vignette  scenarios  were  not  direct  mission,  enemy,  terrain,  troops,  time,  and  civilian 
considerations  (METT-TC)  slices  of  the  larger  scenario,  the  events  that  provided  vignette 
conditions  were  drawn  from  the  scenario.  Figure  3  contains  a  brief  description  of  the  COBRAS  II 
scenario. 


16 


The  Road  to  War 

The  scenario  stoiyline  begins  with  the  brigade  in  a  field  training  exercise  (FTX),  having  been  deployed 
to  the  country  of  Mojave  because  of  an  increasing  threat  from  the  Krasnovians.  The  FTX,  which 
creates  decrements  in  personnel  and  equipment  status,  ends  as  intelligence  indicates  that  the 
Krasnovians  are  preparing  to  cross  the  international  border. 

Note:  The  mission  sequence  presented  below  is  the  recommended  sequence;  however,  the  BSE  will 
support  any  of  the  three  missions  being  nm  in  any  sequence. 

The  Area  Defense  (AD) 

The  brigade  staff  then  receives  an  order  for  an  AD  mission,  and  must  simultaneously  conduct  its 
consolidation  and  reorganization  activities  and  plan  for  the  AD  using  a  modified  decision-making 
process.  The  Krasnovians  conduct  their  attack,  which  fails  after  inflicting  some  losses  to  the  brigade. 
The  brigade  then  conducts  a  rearward  passage  of  lines  to  a  rear  assembly  area  (AA)  (not  performed  in 
the  exercise). 

The  Deliberate  Attack  (DATK) 

Three  days  later  (these  three  days  are  not  represented  in  the  exercise),  the  brigade  receives  a  subsequent 
order  to  conduct  a  DATK  mission  against  the  remnants  of  the  Krasnovians.  The  three  days  in  the 
scenario  provide  a  realistic  time  for  the  brigade  to  return  its  forces  to  a  level  of  combat  readiness  which 
will  make;  the  DATK  mission  feasible.  Again,  the  brigade  staff  uses  the  deliberate  decision-making 
process  (DDMP)  to  develop  its  plan,  briefs  the  plan,  monitors  preparation  activities,  and  conducts  the 
attack.  The  storyline  ends  with  the  defeat  of  the  Krasnovians. 

The  Movement  to  Contact  (MTC) 

While  the  brigade  is  in  its  AA  conducting  medical,  repair,  and  replacement  activities,  it  receives  an 
order  to  conduct  an  MTC  mission.  Using  the  DDMP,  the  brigade  staff  prepares  their  order,  while 
subordinate  units  continue  their  logistics  efforts.  Tire  MTC  is  conducted  against  tlie  Krasnovians’ 
meeting  battle. 

After  the  brigade  has  met  and  defeated  the  Advance  Guard  Main  Body  (AGMB),  circumstances  require 
the  brigade  and  the  enemy  to  stop  their  advances  and  take  up  hasty  defenses.  The  Krasnovian  lead 
elements  retreat  to  hasty  defensive  positions  awaiting  the  arrival  of  second  echelon  forces  to  conduct  an 
attack. 


Figure  3.  Summary  of  COBRAS  II  scenario  storyline. 

Designating  the  Training  Objectives 

Because  of  the  addition  of  new  training  audience  members,  additional  training  objectives 
and  tasks  were  required  to  augment  the  COBRAS  I  lists.  Additional  analysis  was  also  required  to 
determine  how  the  addition  of  the  new  audience  and  scenario  events  would  affect  the  tasks  for  the 
existing  audience  (i.e.,  those  1 1  designated  during  the  COBRAS  I  project).  Together,  these 
requirements  guided  the  COBRAS  11  project's  task  identification  process. 

During  the  initial  months  of  the  project,  the  decision  was  made  to  identify  the  tasks  for  the 
new  training  audience  members  based  on  the  same  considerations  as  were  used  during  the 
COBRAS  I  project:  The  tasks  were  to  be  consistent  with  current  doctrine,  but  could  represent 
performance  requirements  that  were  not  documented  in  the  Army's  doctrinal  publications  or  other 
task-focused  R&D  efforts. 
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The  task  identification  process  was  modeled  on  the  one  used  in  the  COBRAS  I  project. 
The  original  methodology  was  titled  the  Staff  Performance  Analysis  (SPA),  documented  by  Ford 
and  Campbell  (1997);  the  variant  process  for  COBRAS  11  was  referred  to  as  the  modified  staff 
performance  analysis,  or  Modified  SPA  (ModSPA).  The  SPA  used  roleplay  enactments  by 
COBRAS  staff  with  expertise  in  all  aspects  of  brigade  operations  as  the  primary  means  of 
identifying  staff  tasks.  Each  enactment  focused  on  a  single  event,  as  defined  by  logical  break¬ 
points  in  staff  activities.  Most  enactments  required  the  participation  of  the  full  brigade  staff,  or  at 
least  that  part  of  the  staff  represented  by  the  targeted  training  audience. 

The  SPA  provided  opportunities  to  identify  the  tasks  for  all  training  audience  members 
concurrently.  That  is,  as  an  event  was  roleplayed,  every  brigade  staff  representative  was 
documenting  his  or  her  activities  in  preparation  for  further  analyses.  For  the  COBRAS  II 
ModSPA,  a  similar  but  less  time-  and  labor-intensive  method  was  employed.  It  involved  roleplay 
enactments  of  every  scenario  event  (every  event  had  to  be  examined  in  order  to  compile  tasks  for 
the  entire  exercise),  but  did  not  require  the  documentation  of  activities  for  the  entire  brigade  staff. 
The  analysis  would  focus  on  identifying  the  performance  requirements  for  the  new  audience 
members,  but  would  also  serve  to  identify  the  activities  of  other  staff  who  were  involved  in  the 
relevant  aspects  of  the  staff  process.  The  participation  of  the  new  audience  and  the  inclusion  of 
scenario  events  targeting  their  functions  was  expected  to  alter,  to  some  extent,  the  performance 
requirements  for  the  original  training  audience. 

As  a  concluding  verification  of  the  task  lists,  assistance  was  provided  fi-om  job  and  task 
analysts  external  to  the  COBRAS  II  staff  These  analysts  had  been  working  on  the  battlefield 
functions  over  most  of  the  previous  three  years.’  Their  extensive  work  in  exploring  the  staff 
activities  at  brigade  and  battalion  levels  enabled  them  to  provide  a  review  of  the  technical 
accuracy  of  the  draft  task  lists,  to  offer  suggestions  on  modifications  or  additions  to  make  the  lists 
more  descriptive  of  staff  activities,  and  to  provide  a  verification  of  the  consistency  among  lists  for 
various  staff  members. 

Results  of  the  Modified  Staff  Performance  Analysis 

The  ModSPA  yielded  task  lists  for  each  of  the  new  training  audience  members  and  minor 
revisions  to  the  task  lists  of  the  original  training  audience  (described  and  contained  in  Deter, 
Campbell,  &  Ford,  1998).  The  newly  created  task  lists  portray  each  staff  member's  involvement 
and  actions  in  each  mission  segment,  as  well  as  the  actions  and  events  that  should  cue  training 
participants  to  perform  the  required  tasks  (an  example  is  shown  in  Figure  4).  The  task  lists  are 
contained  in  the  BSE  materials  for  two  purposes;  to  assist  observers  in  observation  and  feedback, 
and  to  preview  performance  expectations  for  the  training  audience  members. 


’  The  battlefield  functions  work  is  described  in  Harrison  (1993). 
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1  Task: 
Cue: 
Product: 

2  Task: 


3  Task: 


Product: 


4  Task: 


Product: 

5  Task: 

Product: 

6  Task: 


7  Task: 


Product: 

8  Task: 

9  Task: 


Product: 


Brigade  Operations  and  Training  Oflicer  Task  List 
for  Mission  Analysis-Movement  to  Contact  Mission 

Updates  operations  estimate  with  current  battlefield  information. 

Receipt  of  division  warning  order  (WARNO) 

Updated  operations  estimate 

Briefs  the  current  brigade  status  to  include  locations  of  maneuver  units  and  other  critical 
weapons  systems,  tactical  command  post  (CP),  and  tactical  operations  center  (TOC). 

On  request  of  the  Brigade  Executive  Officer  (XO)  or  Commander 

Reproduces  sufficient  copies  or  extracts  of  division  order  and  distributes  to  Brigade  XO, 
staff,  and  other  key  participants. 

Sufficient  copies  of  the  division  order/extract  for  the  brigade  staff  to  use  in  decision¬ 
making  process 

Reads  and  analyzes  the  Corps  Commander's  intent  in  conjunction  with  the  corps  mission 
and  scheme  of  maneuver. 

Sketch  of  the  corps  scheme  of  maneuver 

Reads  and  analyzes  the  Division  Commander's  intent  in  conjunction  with  the  division 
mission  and  scheme  of  maneuver. 

Sketch  of  the  division  scheme  of  maneuver 

Assesses  task  organization  within  the  division  to  determine  its  possible  effects  on  the 
brigade's  mission  and  tasks. 

Reads  and  analyzes  the  division  order  and  extracts  information  related  to  Army  airspace 
command  and  control  (A2C2)  operations  such  as  A2C2  concept,  A2C2  control 
measures,  suppression  of  enemy  air  defense  operations  in  the  division,  enemy  and 
firiendly  air  corridors,  attack  positions  and  objectives,  forward  arming  and  refueling 
point  locations,  and  identification,  fnend  or  foe/selective  identification  feature  lines. 
Updated  A2C2  estimate 

Identifies  and  assesses  assets  available  to  the  brigade. 

Conducts  terrain  analysis  and  trafficability  in  the  brigade  sector  with  the  Brigade 
Intelligence  Officer  and  Brigade  Engineer  to  include:  avenues  of  approach,  mobility 
corridors,  key  terrain,  and  obstacles. 

Initial  terrain  analysis 


Figure  4.  Sample  of  individual  task  list  resulting  from  the  modified  staff  performance  analysis 


activities. 


Although  these  tasks  contained  information  concerning  staff  interactions,  they  were  not 
directly  usable  in  providing  feedback  on  staff  processes  in  the  form  of  after  action  review  (AAR) 
sessions.  The  AARs  are  generally  group  sessions  involving  all  members  of  the  primary  training 
audience,  in  which  the  observers  facilitate  discussions  of  what  happened,  what  should  have 
happened,  and  what  staff  behaviors  should  be  sustained  or  improved  (DA,  1991).  To  construct 
the  AAR  framework,  the  tasks  were  aggregated  into  staff  process  descriptions,  which  were  then 
prepared  in  the  form  of  AAR  outlines  for  use  by  the  observers.  A  sample  is  shown  in  Figure  5. 
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Mission  Analysis 

•  Prepare  initial  intelligence  preparation  of 

• 

Assess  risk 

the  battlefield 

• 

Briefing  and  approval 

•  Analyze  higher  mission 

•  Analyze  division  order 

Course  of  Action  (COA)  Development 

•  Analyze  commander’s  guidance 

• 

Develop  scheme  of  maneuver 

•  Analyze  relative  combat  power 

• 

Incoiporate  all  BOS 

•  Generate  conceptual  possibilities 

Wargaming 

•  Follow  a  wargame  method 

• 

Record  results 

•  Employ  all  friendly  forces/resources 

• 

Apply  action/reaction/counteraction 

•  Identify  critical  events  and  decision 

• 

Assess  results 

points 

COA  Comparison 

•  Select  comparison  method 

• 

Assign  weighting  values  to  criteria 

•  Determine  decision  criteria 

• 

Record  results  (COA  decision  matrix) 

Orders  Preparation 

•  Incorporate  commander’s  guidance 

• 

Incorporate  BOS  plans 

•  Identify  who,  what,  where,  when,  and 

• 

Review  and  approve 

why 

Rehearsal 

•  Define  the  standard 

• 

Select  key  events  to  rehearse 

•  Orient  participants  to  terrain 

• 

Describe  enemy  CO  As 

•  Verbalize  concept  of  operation 

• 

Focus  on  three  major  purposes  of 
rehearsals 

Modified  Decision-Making  Process 

•  Mission  analysis 

• 

Wargame 

•  Essential  information 

• 

Orders  preparation 

•  Commander’s  concept 

• 

Rehearsal 

Mission  Execution 

•  Acquire  information 

• 

Disseminate  information 

•  Analyze  and  interpret  information 

• 

Make  recommendations  and  decisions 

•  Update  estimates 

• 

Supervise 

Figure  5.  Staff  process  description  formed  from  the  individual  task  lists. 


Because  of  the  low-resource  nature  of  vignettes,  with  no  significant  observer  staff  to  assist 
in  performance,  the  individual  task  lists  were  not  directly  incorporated  in  the  vignettes.  Rather,  as 
with  the  AAR  materials,  descriptions  of  staff  process  pertinent  to  the  vignette  event  were  created. 
These  descriptive  statements  served  as  the  training  objectives,  and  discussion  probes  were  also 
developed  to  present  more  detailed  behavioral  descriptions. 

The  ModSPA  also  facilitated  generation  of  sample  tactical  products  that  would  be  used  as 
training  aids  in  association  with  the  BSE.  Sample  products  were  to  be  used  by  the  observers  to 
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show  the  training  audience  “a-way”  examples  of  the  tactical  products  that  should  result  from  the 
staff  process. 


Designing  the  Exercise  Architecture 

Every  aspect  of  the  original  (COBRAS  I)  BSE  was  precisely  designed  to  support  the 
intent  of  the  exercise,  which  was  driven  by  the  specific  training  audience  and  the  scenario. 
Participants  had  well  defined  and  purposeful  roles,  simulation  layout  and  usage  were  described  in 
specific  terms,  the  scenario  was  highly  structured,  and  the  TSP  was  thorough  and  direct.  Because 
the  exercise  was  not  intended  to  train  all  aspects  of  brigade  functioning,  the  model  specified  8-10 
hour  workdays  and  operations  out  of  administrative  CP  locations  in  or  near  the  simulation  center. 
Thus,  all  components  of  the  design  and  the  TSP  itself  were  created  to  represent  a  tightly  linked 
group  of  products  that,  when  put  together,  would  support  the  functioning  of  all  participants. 

In  the  COBRAS  II  effort,  even  though  the  same  basic  exercise  model  was  to  be  employed, 
the  training  audience  was  expanded,  leading  to  a  sequence  of  changes  throughout  the  exercise's 
architecture.  Additionally,  lessons  learned  during  the  development  of  the  original  BSE  had  to  be 
incorporated.  The  changes  to  the  BSE  structure  fall  into  three  areas. 

•  participant  types  and  numbers, 

•  roleplayer  and  interactor  training,  and 

•  performance  observation  and  feedback. 

Participant  Types  and  Numbers 

The  number  of  participants  changed  not  only  by  the  addition  of  the  new  training  audience 
members,  but  also  because  of  the  need  for  supporting  personnel  for  those  functions  (i  .e.  , 
observers,  roleplayers,  and  interactors).  Overall,  the  increase  was  from  1 13  to  132  participants. 

Concerning  observers,  the  COBRAS  I  BSE  had  recommended  a  staffing  plan  that 
combined  some  of  the  training  audience  members  functionally  under  observers.  The  revised  BSE, 
however,  specified  one  observer  for  each  training  audience  member;  a  one-for-one  staffing 
arrangement.  As  a  compromise  to  support  circumstances  where  obtaining  16  observers  might  be 
unfeasible,  the  new  BSE  also  offered  a  less  intensive  staffing  plan  similar  to  the  plan  proposed  in 
the  original  BSE.  This  minimal  staffing  plan  called  for  ten  observers.  New  observer  positions 
included  a  Chemical  Observer,  a  Signal  Observer,  a  Personnel  and  MP  Observer,  an  Army 
Aviation  Observer,  an  Engineer  Observer,  and  an  air  defense  artillery  (ADA)  Observer.  The  last 
two  of  these  split  the  duties  of  what,  in  the  COBRAS  I  BSE,  had  been  the  CS  Observer. 

Four  roleplayers  were  added  to  provide  information  and  task  cues  to  the  new  target 
audience.  Additionally,  five  interactors  were  added  to  staff  the  simulation  workstations.  Figure  6 
presents  the  COBRAS  II  BSE  exercise  layout. 
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Exercise  Director 
COBRAS  Coordinator 


HICON  Workstation: 
HICON  Workstation  Operator/ 
BBS  System  Controller 


EXCON  Workstation: 
DIV  G3.  G2.  G1/4.  FSO,  ADA, 
ENG,  AVN  Roleplayers; 

3  Interactors;  IRTO/Admin 


ii^ 


Target  Training  Audience 

Brigade  Main  CP: 

Bde  Cdr,  XO,  S2,  S3,  FSO, 
FSCOORD,  ADCOORD, 

ENG,  SIGO,  CHEMO,  Ml  Co 
Cdr,  Avn  LNO,  MP  Pit  Ldr 

Brigade  TAC:  (execution  only) 
Bde  Cdr,  S3,  FSCOORD 

Brigade  Rear  CP: 

S1,  S4,  FSB  Cdr 

Assistant  staff  officers  and  NCOs 


P 


Observers: 

One  per  target  audience  member  (16) 
-or- 

Senior  Observer.  Ops.  Intel, 

FS,  Personnel  and  MP,  CSS, 

ENG,  ADA,  Chemical,  Signal, 
Army  Aviation 


OPFOR  Workstations  (3): 

OPFOR  Controller; 

2  Interactors  per  station 


Blue  Forces  Controller 


Task  Force  Workstations  (4): 

Ops,  Intel,  CSS,  CS  Roleplayers; 

3  Interactors;  1  RTO/Clerk 


Cavalry  Troop  Workstation: 

Ops/CS,  CSS  Roleplayers; 

3  Interactors;  1  RTO/Clerk 


FA  Battalion  Workstation: 
Ops/FC,  CSS,  Intel  Roleplayers; 
3  Interactors;  1  RTO/Clerk 


Engineer  Battalion  Workstation: 
Ops/CS,  CSS,  2  ENG  Co  Cdr  Roleplayers; 
3  Interactors;  1  RTO/Clerk 


ADA  Battery  Workstation: 

Ops/CS,  css  Roleplayers; 

3  Interactors;  1  RTO/Clerk 


FSB  Workstation: 

Ops,  css  Roleplayers; 

3  Interactors;  1  RTO/Clerk 


Brigade  Troops  Workstation: 

Ml,  MP,  Chemical  Roleplayers; 

3  Interactors;  1  RTO/Clerk 


Figure  6.  Participant  locations  during  conduct  of  a  COBRAS  II  Brigade  Staff  Exercise. 
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Roleplaver  and  Interactor  Simulation  Training 

In  order  to  facilitate  BBS  training  for  interactors  and  roleplayers,  the  COBRAS  I  BSE 
provided  a  brief  training  outline  and  slides.  However,  during  the  COBRAS  I  pilot  and  trial 
implementation,  the  training  for  interactors  and  roleplayers  was  not  sufficiently  detailed  on  the 
specific  functions  to  be  performed  during  the  COBRAS  program  (Graves  et  al.,  1997). 

In  the  development  of  the  COBRAS  n  BSE,  the  developers  chose  to  provide  more 
specific  guidance  regarding  the  BBS  training  needs  that  were  required  by  the  BSE.  The 
COBRAS  BBS  roleplayer/interactor  training  plan  was  designed  and  written  to  supplement  the 
interactor  training  that  a  site  typically  provides.  The  program  focuses  on  helping  the  interactors 
and  roleplayers  to  implement  the  COBRAS  scenario  in  BBS,  an  activity  that  is  key  to  allowing  the 
brigade  staff  to  achieve  their  training  objectives. 

The  roleplayer/interactor  training  is  a  three-phase  program,  including  BBS  train-up  for 
interactors,  practical  exercises  in  which  interactors  practice  the  tasks  that  are  important  during 
BSE  execution,  and  a  mini-exercise  in  which  interactors  and  roleplayers  can  learn  how  to  work 
together  to  accomplish  their  objectives  and  portray  the  scenario.  The  role  of  the  site  staff  is  to 
help  interactors  learn  how  to  operate  BBS  and  to  help  roleplayers  learn  how  to  work  with 
interactors. 

The  roleplayer/interactor  training  is  executed  according  to  the  guidance  provided  in  the 
BBS  Site  Manager’s  Guide,  and  uses  two  other  information  sources  as  well;  the  BBS  5.0 
Warfighter's  Guide  (National  Simulation  Center;  BBS  General  Support  Team;  Logicon  Technical 
Services,  Inc.,  and  U.S.  Army  Communication  Electronics  Command,  1994)  and  the  COBRAS 
interactor  guides.  The  program,  described  below,  was  thoroughly  documented  in  the  COBRAS 
BBS  Site  Manager’s  Guide,  a  component  of  the  BSE  TSP  that  was  added  for  COBRAS  II. 

The  overall  goal  of  the  roleplayer/interactor  training  is  to  establish  a  good  working 
relationship  between  roleplayers  and  interactors;  roleplayers  should  know  how  to  utilize  the 
interactors.  Specific  objectives  for  each  phase  of  the  training  are  described  in  the  COBRAS  BBS 
Site  Manager’s  Guide  as: 

•  Phase  1  -  “...interactors  should  be  familiar  with  the  scope  and  specifics  of  the  tasks 
they  are  to  perform.  They  should  also  be  able  to  execute  the  most  common  and 
important  tasks  on  the  simulation.” 

•  Phase  2  -  “...interactors  should  have  improved  their  ability  to  perform  BBS  tasks  well 
enough  to  conduct  a  mini-exercise  with  roleplayers.” 

•  Phase  3  -  “...the  interactors  should  be  able  to  hone  their  BBS  skills  and  the  roleplayers 
should  become  generally  familiar  with  BBS.” 

By  the  end  of  the  mini-exercise,  all  of  the  interactors  and  roleplayers  should  have  some 
familiarity  with  how  to  execute  the  mission  in  BBS,  as  well  as  a  common  understanding  of  the 
differences  between  BBS  and  tactical  terminology.  Additionally,  roleplayers  must  understand 
which  operations  and  combat  functions  can  and  cannot  be  performed  in  BBS  and  which 
operations  have  to  be  performed  by  magic  at  the  higher  control  workstation. 
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Performance  Observation  and  Feedback 

Significant  modifications  were  made  to  the  BSE  design  in  the  area  of  performance 
observation  and  the  provision  of  feedback.  The  role  of  the  observers  in  the  COBRAS  II  BSE  was 
essentially  the  same  as  in  COBRAS  I:  to  serve  as  mentors,  coaches,  and  providers  of  feedback 
both  in  informal  discussions  and  in  more  formal  AARs  with  the  brigade  staff.  In  COBRAS  I, 
fi-equent  AARs  (at  least  daily)  were  used  instead  of  the  more  common  model  of  an  AAR  only  at 
mission  completion.  This  approach  was  followed  again  in  COBRAS  11.  However,  as  a  result  of 
observations  during  the  Fort  Riley  COBRAS  I  trial  implementation,  certain  modifications  were 
made  in  the  observer  TSP  materials. 

One  of  these  changes  concerned  AAR  preparation.  During  the  trial  implementation, 
observers  remarked  that  there  was  not  enough  time  in  the  schedule  to  prepare  for  the  Ai^,  and 
requested  a  two  hour  pause  in  the  exercise  (one  hour  to  prepare,  one  hour  for  the  AAR  itself).  At 
the  same  time,  training  audience  members  stated  that  one  hour  of  “pause  time”  in  the  exercise  for 
AARs  was  as  much  as  they  were  willing  to  give  up.  As  a  compromise,  the  instructions  to 
observers  were  modified  to  suggest  that  they  take  an  hour  for  AAR  preparation  while  the  brigade 
staff  continued  to  work  on  the  next  phase  of  the  exercise;  the  AAR  itself  would  still  require  only 
one  hour  of  pause. 

Another  change  was  made  to  the  AAR  materials  themselves.  More  slides  were  added  to 
the  package  to  provide  the  observers  with  additional  visual  presentations  of  the  critical  materials. 
Most  of  the  added  slides  were  expected  to  serve  as  back-up  material,  in  case  the  brigade 
commander  wanted  additional  discussion  on  specific  points.  As  with  COBRAS  I,  the  slides  were 
presented  both  on  paper  and  in  a  Microsoft®  PowerPoint®  format.  The  electronic  version  was 
provided  to  enable  observers  to  customize  the  slide  content  and  order  for  their  own  use. 

The  AAR  focus  for  each  mission  segment  was  also  revised  in  order  to  clarify  the  intent. 
Each  AAR  was  focused  on  helping  the  brigade  commander  and  his  staff  determine  if  the  outputs 
of  that  segment  would  provide  all  that  was  needed— the  inputs— for  the  next  segment.  These 
outputs  could  be  products,  guidance,  decisions,  staff  input,  information,  or  analysis  and  sharing  of 
information.  The  AARs  were  designed  to  help  guide  the  staff  to: 

•  look  at  what  they  just  accomplished  and  the  processes  they  used, 

•  compare  it  to  what  they  intended  to  accomplish  and  the  processes  they  intended  to 
use, 

•  determine  if  they  needed  to  make  changes  to  their  processes,  and 

•  describe  how  they  intended  to  accommodate  those  changes  in  future  operations. 

The  third  area  of  change  concerned  the  provision  of  the  sample  brigade  products.  These 
were  to  be  used  to  show  or  demonstrate  selected  outcomes  of  staff  activities.  A  selected  set  of 
products  was  reviewed  during  the  trial  implementation  with  the  observers,  and  their  reactions 
were  uniformly  positive.  These  products  were  developed  during  the  ModSPA  described  above. 
They  were  intended  to  serve  as  models  of  type  of  content  only,  as  the  specific  contents  would 
depend  on  the  particular  decisions  worked  out  by  the  brigade  staff  during  their  planning. 

Finally,  there  was  a  more  explicit  emphasis  on  involving  the  commander  in  designing  the 
specific  observation  plan.  The  Senior  Observer  was  instructed  to  meet  with  the  brigade 


24 


commander  to  clarify  his  (the  commander’s)  expectations,  paying  particular  attention  to  his  stated 
training  objectives  and  focus  so  that  the  observer  team  could  be  attuned  to  that  emphasis.  The 
Senior  Observer  was  to  review  the  unit  tactical  standing  operating  procedures  (TACSOP)  and 
resolve  any  differences  between  it  and  the  COBRAS  task  lists  and  AAR  materials  by  means  of 
discussions  with  the  commander.  Additionally,  all  of  the  observers  were  expected  to  become 
familiar  with  the  TACSOP  before  the  exercise  began. 

Developing  the  Training  Support  Package 

As  the  changes  in  the  participants  and  other  aspects  of  the  BSE's  design  were 
incorporated,  the  BSE  TSP  changed  accordingly.  Because  the  trial  implementation  would  involve 
only  one  mission,  development  focused  on  design  for  that  mission  only.  The  major  changes  that 
were  made  in  preparation  for  the  trial  implementation  were  as  follows; 

•  The  tactical  materials  were  revised  as  described  above,  providing  the  modified  division 
OPORDs,  intelligence  reports,  and  message  traffic  for  the  added  scenario  events. 
Several  additional  overlays  were  also  provided  because  of  the  aviation  events  now 
being  included. 

•  The  training  audience  preparation  materials  for  the  new  TSP  contained  a  more 
complete  description  of  the  AAR  process. 

•  Observer  materials  also  contained  the  fuller  AAR  description,  as  well  as  guidance  on 
the  use  of  the  sample  products.  A  completely  new  set  of  AAR  materials  was  provided, 
with  more  slides  (especially  back-up  slides)  and  textual,  guidance  for  the  slides.  The 
sample  products  were  included  in  a  separate  package. 

•  The  interactor  guides  were  restructured  to  provide  separate  materials  for  each  type  of 
BBS  workstation  terminal  (i.e.,  combat,  CS,  and  CSS),  because  the  activities  at  the 
terminals  vary  so  widely. 

•  Additional  roleplayer  guides  were  prepared  to  cover  the  separation  of  the  ADA  and 
engineer  functions  as  well  as  the  addition  of  the  brigade  troops  workstation. 

•  The  OPFOR  Controller  was  given  somewhat  more  latitude  in  choosing  schemes  of 
maneuver,  always  under  the  oversight  of  the  Exercise  Director. 

•  Instructions  on  conduct  of  the  exercise  in  the  Exercise  Guide  were  modified  to  track 
with  the  changed  scenario.  Instructions  on  TSP  distribution  were  modified  in 
accordance  with  revisions  to  the  TSP  and  were  also  expanded  in  order  to  make  them 
easier  to  use. 

•  The  BBS  Table  of  Organization  and  Equipment  (TOE)  and  Initialization  Book  and 
the  BBS  Archive  Book,  which  contain  documentation  on  the  BBS  database 
specifications,  were  revised  to  enhance  clarity  and  completeness. 

•  Three  new  guides  were  added  to  the  TSP  for  the  COBRAS  II  BSE: 

o  The  Brigade  Orientation  Guide  provides  the  brigade  commander  with  a  complete 
description  of  the  exercise  and  its  resource  requirements.  It  is  designed  to  help  the 
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commander  decide  whether  or  not  he  should  implement  the  COBRAS  II  BSE,  and 
what  he  needs  to  do  to  get  started. 

o  The  XO  Guide  contains  instructions  for  the  brigade  XO  on  staff  preparation  for  the 
exercise,  and  also  describes  how  CSS  reports  have  been  distributed  to  the  training 
audience  members  in  their  preparation  materials.  This  last  information  allows  the 
XO  to  know  what  information  about  starting  conditions  has  been  provided  to 
various  elements  of  the  training  audience. 

o  The  Simulation  Site  Manager ’s  Guide  includes  descriptions  of  the  BBS  tapes  and 
other  documentation.  The  intent  was  to  inform  the  simulation  site  staff  about  the 
rationale  for  the  structure  and  contents  of  the  database,  so  that  they  would  not  feel 
compelled  to  revise  the  database,  and  so  that  any  revisions  requested  by  the  brigade 
commander  could  be  made  without  damaging  the  integrity  of  the  exercise. 

Figure  7  provides  a  description  of  the  overall  COBRAS  II  BSE  TSP  structure,  as  it 
existed  going  into  the  trial  implementation. 


Summary 

Even  with  the  COBRAS  I  experience  and  materials,  development  of  the  COBRAS  11  BSE 
and  vignettes  was  a  complex  process.  The  initial  development  described  here  resulted  in 
preparation  of  the  BSE  and  vignette  exercises  and  TSPs  ready  for  trial  implementation.  The  next 
section  (Section  4)  describes  the  BSE  trial  implementation  and  evaluation  plan,  the  findings,  and 
the  results  of  that  trial,  including  the  revisions  made  to  the  exercise  and  TSP.  For  vignettes,  the 
evaluation  processes  will  be  described  in  Section  5. 
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TSP  Category ' 


Exercise 

Management 

•  Exercise  Guide  for  the  Exercise  Director,  COBRAS  Coordinator,  and  Blue 
Forces  Controller,  with  Appendixes 

•  Brigade  Orientation  Guide 

Tactical  Materials 

•  Corps  Concept  (Movement  to  Contact,  Area  Defense,  and  Deliberate  Attack) 

•  Division  Orders  and  Tactical  Materials  (including  overlays) 

•  Scripted  and  hard-copy  messages 

Participant  Guides 
and  Materials 

Training  Audience.’ 

•  Training  Audience  Guide  (generic,  for  all  16  training  audience  members) 

•  XO  Guide  to  Unit  Preparation  and  Materials  Distribution 

•  Initial  Situation  Packages  and  start  of  exercise  (STARTEX)  Position  Overlays 
(per  staff  member,  per  mission) 

•  Task  Lists  (per  staff  member,  per  mission) 

Observers.’ 

•  Observer  Guide  (generic,  for  all  Observers) 

•  Task  Lists  (per  Observer,  per  mission) 

•  Observer  After  Action  Review  Briefing  Materials 

Roleplayer  Teams: 

•  EXCON  Roleplayer  Guide  •  Opposing  Force  Controller  Guide 

•  TF  1-5  Roleplayer  Guide  •  Cavalry  Troop  Roleplayer  Guide 

•  TF  1-7  Roleplayer  Guide  •  Field  Artillery  Roleplayer  Guide 

•  TF  3-5  Roleplayer  Guide  •  Engineer  Roleplayer  Guide 

•  TF  1-80  Roleplayer  Guide  ,  PSB  Roleplayer  Guide 

•  ADA  Roleplayer  Guide  ,  Brigade  Troops  Roleplayer  Guide 

•  Initial  Situation  Packages  and  STARTEX  Position  Overlays  (per  roleplayer 
team,  per  mission) 

BBS  Interactors: 

•  EXCON  Interactor  Guides  (versions  for  CS,  and  CSS  Interactors  at  EXCON 
workstation) 

•  Blue  Interactor  Guides  (versions  for  Combat,  CS,  and  CSS  Interactors  at  Blue 
workstations) 

•  Red  Interactor  Guides  (versions,  for  Combat  and  CS  Interactors  at  Red 
workstations) 

Simulation 

Materials 

Tools  for  initializing  BBS  and  making  changes  or  corrections: 

•  Simulation  Site  Manager’s  Guide 

•  BBS  TOE  and  Initialization  Book 

•  BBS  Archive  Book 

•  BBS  System  Tapes 

Figure  7.  Structure  of  the  COBRAS  II  Brigade  Staff  Exercise  training  support  package  at  the 
time  of  the  trial  implementation. 
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SECTION  4:  TRIAL  IMPLEMENTATION  OF  THE  BRIGADE  STAFF 

EXERCISE 


The  key  evaluation  event  during  the  development  of  the  COBRAS  H  BSE  was  the  tnal 
implementation  conducted  at  Fort  Lewis,  WA,  with  S'"  Brigade,  2  ID.  This  trial  involved  use 
the  TSP  for  one  mission  with  participation  of  personnel  representative  of  the  target  trairang 


of 


audience. 

This  section  describes  that  trial,  and  indicates  the  evaluation  methodology  employed  and 
the  trial  results.  Three  topics  are  discussed; 

•  Trial  Implementation:  Describes  the  specifics  of  the  formative  evaluation  in  terns  of 
the  purpose  of  the  trial,  the  evaluation  methods  and  restrictions/constramts,  and  t  e 
evaluation  staffing  plan. 


•  Trial  Implementation  Results:  The  trial  results  represent  analysis  of  feedback 
provided  by  participating  unit  members  and  observations  made  by  COB^S 
developers.  The  findings  provide  a  comprehensive  summary  portrayal  of  the  data 

collected. 


•  Program  Revisions:  Describes  the  extent  of  revisions  that  were  made  to  the  TSP  as  a 
result  of  the  trial. 


Trial  Implementation 

The  purpose  of  the  trial  implementation  was  to  permit  evaluation  of  the  effect  of  traimng 
program  design  decisions,  the  utility  of  the  TSP  components,  and  perceptions  of  traimng  benefits. 
Exercise  developers  observed  the  planning,  preparation,  and  conduct  of  the  exercise  and 
performed  various  data  collection  activities.  Although  developers  were  prepared  to  assist  m 
implementation  as  necessary,  in  order  to  ensure  that  the  exercise  could  be  conducted,  the  intent 
was  to  operate  in  a  “hands-off’  mode  as  much  as  possible. 

Certain  features  of  the  actual  implementation  limited  the  ability  to  generalize  the  findings 
beyond  the  single  implementation.  In  general,  the  exercise  was  implemented  accoirimg  to  the 
guidance  provided  in  the  TSP.  There  were  several  departures,  however,  as  desenbed  below. 

.  Because  only  one  mission  (the  DATK)  was  exercised,  CSS  activities  were  not  fully 
performed.  These  activities  generally  take  place  between  missions,  when  the  brigade  is 
recovering  from  one  operation  and  preparing  for  the  next. 

•  The  unit  employed  three  task  force  (TF)  units  and  no  cavalry  troop  during  the 
exercise,  rather  than  the  four  TFs  and  one  cavalry  troop  called  for  in  the  scenano. 
While  tWs  organization  matched  the  brigade’s  actual  structure,  the  change  required 
modification  of  the  division  OPORD,  brigade  sector,  and  associated  tactical  matenals. 
Thus  evaluation  of  the  exact  METT-TC  elements,  as  designed,  was  incomplete. 
However,  the  trial  implementation  still  produced  many  valuable  insights  concerning 
tactics,  unit  desires,  and  the  complexity  of  correctly  generating  all  of  the  cues  needed 
for  unit  performance. 


28 


•  The  observers  had  little  brigade  level  experience,  although  they  did  have  extensive 
observer  experience  at  the  battalion  level. 

•  The  model  called  for  a  senior  officer,  preferably  from  the  brigade’s  higher 
headquarters,  to  serve  as  the  Exercise  Director.  However,  because  no  one  of  that 
level  was  available  to  fill  the  role,  those  duties  and  responsibilities  were  shared  by  the 
brigade  commander,  the  Senior  Observer,  and  the  officer  in  charge  (OIC)  of  the 
simulation  center,  with  some  assistance  from  the  COBRAS  team  members  on  site. 

•  Pre-exercise  training  conducted  by  the  simulation  site  staff  did  not  follow  the 
roleplayer/interactor  training  plan  contained  in  the  TSP.  Rather,  the  staff  used  their 
own  familiarization  approach.  This  change  was  estimated  to  have  had  little  negative 
effect  on  acquisition  of  BBS  skills,  but  did  it  not  provide  the  additional  COBRAS 
information  and  emphasis.  Observations  of  the  training  were  used  to  modify  the 
COBRAS  training  description  to  bring  it  more  in  line  with  what  was  actually  done  and 
with  what  should  be  anticipated  at  other  BBS  sites. 

COBRAS  staff  involvement  in  the  implementation  was  extensive.  Two  pre¬ 
implementation  trips  were  conducted  to  coordinate  the  staffing  for  the  exercise,  work  with  the 
simulation  site  staff  on  modifying  the  BBS  files,  and  assist  in  modifications  to  the  tactical 
materials.  Innumerable  telephone  conversations  occurred  to  ensure  that  the  TSP  materials  were 
being  correctly  assembled  and  distributed.  In  every  case  where  COBRAS  staff  assisted  in 
preparations,  detailed  notes  were  recorded  about  the  nature  of  the  request  for  assistance,  so  that 
the  TSP  could  be  examined  and  revised  as  necessary. 

For  the  conduct  of  the  trial  implementation  itself,  12  of  the  COBRAS  staff  were  on  site 
before  the  interactor  and  roleplayer  training  began,  two  days  prior  to  the  exercise  start.  Again  any 
requests  for  assistance  were  noted  for  later  incorporation  into  the  materials. 

Trial  implementation  data  were  collected  by  means  of  four  general  methods: 

•  Structured  interviews  were  conducted  with  key  participants,  including  all  three 
members  of  the  exercise  administrators  (the  simulation  center  OIC,  COBRAS 
Coordinator,  and  Blue  Forces  Controller),  all  observers  and  training  audience 
members,  the  individual  representing  the  Division  Operations  Officer  (G3)  at  EXCON, 
and  the  OPFOR  Controller.  The  interview  protocols  had  been  prepared  before  the 
trial  began,  in  order  to  ensure  that  careful  attention  was  paid  to  critical  evaluation 
issues. 

•  COBRAS  developers  observed  every  phase  of  the  implementation  process,  from  initial 
unit  decisions  and  preparation  and  distribution  of  the  TSP  materials  through  conduct 
of  the  exercise  itself 

•  Questionnaires  were  prepared  and  distributed  to  all  of  the  participants,  including  even 
the  brigade  assistant  staff  members  and  staff  sections,  all  roleplayers,  and  all 
interactors.  The  questionnaires  contained  a  mixture  of  two  types  of  items:  5-point 
scale  response  items  (e.g.,  disagree-agree)  and  open  ended  items  (e.g.,  what  additional 
initialization  instructions  were  needed?). 
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•  Group  discussions  at  the  conclusion  of  the  exercise  were  used  to  obtain  more  in-depth 
information  and  perceptions  of  the  training  quality  and  value,  and  ideas  about  exercise 
features  that  should  be  strengthened,  modified,  or  dropped. 

Trial  Implementation  Results 

The  results  fi'om  the  participant  interviews  and  questionnaires  at  the  trial  implementation 
of  the  COBRAS  11  BSE  are  described  below.  The  presentation  is  organized  into  three  topics; 
exercise  design,  TSP,  and  training  benefit. 

Exercise  Design  Findines 

The  aspects  of  the  exercise  design  that  were  evaluated  during  the  trial  implementation 
included  the  following: 

•  resource  requirements, 

•  the  scenario, 

•  brigade  staff  task  lists  and  sample  products, 

•  adequacy  of  staffing  and  experience  specifications, 

•  pre-exercise  BBS  training  for  interactors  and  roleplayers, 

•  observation  and  feedback  procedures, 

•  exportability,  and 

•  implementation  conditions. 

Resource  requirements.  Resource  requirements  are  a  long-standing  concern  in 
simulation-based  exercises.  Given  the  resources  that  brigades  are  able  to  commit  to  training,  the 
exercise  was  designed  to  use  personnel  efficiently,  and  the  TSP  was  organized  to  limit  reading  and 
study  prerequisites  to  the  absolute  minimum.  Of  particular  interest  was  the  extent  to  which  the 
BSE  TSP  would  reduce  the  effort  a  brigade  and  the  simulation  center  staff  would  expect  to 
expend  in  designing,  planning,  preparing  for,  and  conducting  such  an  exercise.  Results 
concerning  resource  requirements  included: 

•  The  simulation  center  OIC  and  COBRAS  Coordinator  were  asked  about  the  extent  to 
which  the  scenario  and  the  implementation  instructions  reduced  preparation  time.  Their 
responses  to  both  questions  were  positive,  indicating  that  the  scenario  and  TSP  were 
valuable  in  reducing  the  total  time  spent  in  preparation  for  the  exercise.  This  finding 
reinforces  the  assumption  that  the  structured  training  concept  is  valuable  for  time- 
strapped  organizations. 

•  Roleplayers  and  interactors  were  asked  to  evaluate  the  time  and  effort  expenditures  for 
the  BSE,  contrasting  those  expenditures  with  those  required  for  other  types  of  training. 
As  shown  in  Figure  8  both  roleplayers  and  interactors  characterized  their  responsibilities 
in  the  BSE  as  being  relatively  less  intensive,  in  terms  of  the  time  and  effort  required, 
than  typical  BBS  exercises. 
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Ratings  of  time  and  effort  required  by  the  BSE 
compared  to  typical  BBS  exercises 


■  More 


■  About  the  Same 
□  Less 


Roleplayers  Interactons 

(minus  EXCX)N)  (n=17) 

(n=16) 


Figure  8.  Feedback  on  time  and  effort  requirements  for  roleplayers  and  interactors. 


•  The  treuning  audience  members  (n=16)  reported  that  the  BSE  required  less  of  an 

investment  by  the  brigade  than  do  typical  BBS  exercises,  leader  training  programs,  and 
NTC  rotations.  Figure  9  presents  the  ratings  provided  for  each  program  type. 


Training  audience  ratings  of  time  and  effort  required 
by  the  BSE  and  other  types  of  staff  training 


Leader  Training  National  Training  BBS  Exercise  COBRAS 

Program  Center 


Figure  9.  Feedback  on  the  time  and  effort  requirements  for  training  audience  members. 

•  The  three  exercise  management  participants  were  asked  how  the  exercise  compared  to 
other  exercises  in  terms  of  personnel  requirements.  There  was  no  consensus  among 
them.  The  simulation  center  OIC  indicated  that  the  BSE  required  more  personnel  than 
other  exercises,  while  the  COBRAS  Coordinator  believed  the  BSE  required  fewer 
persons.  The  Blue  Forces  Controller  reported  that  there  was  no  difference. 

In  sum,  trial  feedback  indicated  that  the  BSE  was  acceptable,  although  not  ideal,  in  terms 
of  the  time  and  personnel  resources  it  requires.  However,  no  major  changes  that  could  have 
significantly  reduced  the  requirements  while  preserving  the  instructional  content  necessary  to 
support  the  training  structure  were  indicated. 

Scenario.  The  scenario  was  another  topic  for  evaluation.  The  COBRAS  II  scenario,  a 
modification  of  the  original  COBRAS  I  scenario,  underwent  numerous  modifications  during  the 
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course  of  the  project.  Map  exercises,  OPORD  reviews,  and  internal  pilots  all  greatly  facilitated 
the  iterative  process  of  designing,  testing,  and  revising  the  scenario.  The  trial  represented  the  final 
test  of  the  scenario  to  verify  the  internally-tested  design  modifications.  Feedback  was  collected 
fi’om  participants  on  various  aspects  of  the  scenario. 

•  The  training  audience,  simulation  center  OIC,  and  COBRAS  Coordinator  were  asked 
about  the  extent  to  which  the  scenario  presented  a  sufficiently  challenging  tactical 
problem  to  drive  planning  and  preparation.  Feedback  indicated  that  the  tactical 
problem  was  moderately  to  highly  challenging  for  the  training  audience  respondents 
(n=16)  (see  Figure  10).  The  simulation  center  OIC  and  COBRAS  Coordinator  also 
responded  that  the  tactical  problem  challenged  the  brigade  staff  members. 


Training  audience  responses  on  the  extent  to  which 
the  scenario  presented  a  challenging  tactical  problem 


Very  Great  Extent  Great  Extent  Moderate  Extent  Slight  Extent 


Figure  10.  Feedback  on  the  difficulty  of  the  tactical  problem. 

•  The  same  participants  were  also  asked  about  the  extent  to  which  the  scenario 

conditions  supported  a  challenging  exercise.  This  item  differs  from  the  previous  one  in 
its  focus  on  the  conditions  during  execution,  as  caused  by  the  OPFOR,  division 
directives,  CSS  conditions,  and  so  on.  The  results  from  the  training  audience  (Figure 
1 1)  were  positive,  as  were  results  from  the  simulation  center  OIC  and  COBRAS 
Coordinator,  indicating  that  the  exercise  was  appropriately  challenging  but  not 
overwhelming. 


32 


Training  audience  responses  on  the  extent  to  which 
the  scenario  supported  a  challenging  exercise 


Very  Great  Extent  Great  Extent  Moderate  Extent 


Figure  11.  Feedback  on  the  difficulty  of  the  exercise  caused  by  scenario  conditions. 

•  Another  aspect  for  scenario  evaluation  concerned  the  completeness  of  CSS  activity  in 
the  BSE,  compared  to  other  BBS  exercises.  Results  from  those  who  were  questioned, 
including  the  training  audience  (n=15),  simulation  center  OIC,  COBRAS  Coordinator, 
and  Blue  Forces  Controller,  indicated  a  dramatic  difference  in  favor  of  the  BSE.  This 
provided  support  for  achievement  of  one  of  the  program  objectives,  to  provide  CSS 
practice  for  the  brigade  staff.  Results  for  the  training  audience  are  shown  in  Figure  12. 


Training  audience  ratings  of  the  amount  of 
CSS  Activity  in  the  BSE  and  in  typical  BBS  exercises 


More  Equal  Less 


Figure  12.  Feedback  on  the  amount  of  combat  service  support  activity  in  the  Brigade  Staff 
Exercise. 

•  The  EXCON  roleplayers  (n=6)  and  OPFOR  controller,  who  were  most  responsible  for 
driving  the  scenario  and  providing  the  performance  cues,  were  asked  about  the  extent 
to  which  the  scenario  cued  opportunities  for  the  training  audience  to  practice  the 
training  objectives.  The  results  were  clearly  positive,  with  six  respondents  reporting 
that  the  scenario  provided  the  required  cues  to  a  great  extent,  and  one  reporting  a 
moderate  extent  of  agreement  that  the  cues  are  appropriate  for  the  training  objectives. 
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•  The  OPFOR  Controller  was  asked  whether  or  not  the  OPFOR’s  specified  courses  of 
action  were  too  restrictive.  His  response  was  ambivalent,  indicating  that  he  neither 
agreed  nor  disagreed  that  the  scenario  was  too  restrictive. 

Brigade  staff  task  lists  and  sample  products.  The  BSE  provides  exercise  specific  task 
lists,  as  well  as  sample  products,  for  the  brigade  staff  and  observers.  A  close  examination  of  the 
utility  of  these  products,  therefore,  was  a  key  component  of  the  evaluation.  Questionnaire  items 
and  interviews  with  observers  and  training  audience  members  were  designed  to  address  utility 
issues.  Five  items  concerning  the  task  lists  and  sample  products  were  posed  to  the  observers. 
The  results  on  the  five  items  were  generally  positive,  indicating  that  both  the  task  lists  and  the 
sample  products  were  useful  to  observers  for  their  intended  purposes.  The  items  are  shown 
below,  with  results  portrayed  in  Figure  13  and  Figure  14. 

•  To  what  extent  did  reviewing  the  task  lists  help  you  prepare  for  the  BSE? 

•  To  what  extent  were  the  task  lists  useful  in  keeping  track  of  what  to  observe? 

•  To  what  extent  were  the  task  lists  useful  in  coaching  the  brigade  staff? 

•  To  what  extent  did  studying  the  sample  products  help  you  prepare  for  the  BSE? 

•  To  what  extent  were  the  sample  products  useful  in  coaching  the  brigade  staff? 


Figure  13.  Feedback  from  observers  on  the  value  of  the  brigade  staff  task  lists. 


Observer  ratings  on  the  utility  of  the  sample  products 


Useful  for  Preparation  (n=6)  Useful  for  Coaching  (n=8) 


Figure  14.  Feedback  from  observers  on  the  utility  of  the  sample  products. 

Three  questionnaire  items  that  focused  on  the  task  lists  and  sample  products  were 
addressed  to  members  of  the  training  audience.  Again,  the  results  were  consistently  positive.  The 
items  are  shown  below,  with  results  shown  in  Figure  15. 

•  To  what  extent  did  the  task  lists  clarify  what  you  would  be  doing  during  the  BSE? 

•  To  what  extent  did  the  task  lists  match  the  demands  of  your  job  during  the  BSE? 

•  To  what  extent  were  the  task  lists  useful  in  helping  you  prepare  for  the  exercise? 

Comments  provided  by  the  observers  and  training  audience  were  also  helpful  in  explaining 
how  the  task  lists  and  products  could  be  used  in  the  BSE  context.  These  comments  included  the 
following; 

•  “The  task  lists  served  as  a  memory  jogger/reminder  as  I  prepared  [to  perform]  each 
event.” 

•  “Task  list  is  a  good  tool  to  review  after  and  before  a  task  is  completed.” 

•  “The  task  lists  are  very  good  for  someone  who  hasn’t  had  much  experience  [as  an 
observer].  It  helps  to  re-trigger  their  minds  on  where  to  focus  when  observing  the 
brigade  staff.” 

•  “Task  lists  are  the  most  important  part  for  [observer]  preparation  and  coaching.” 

•  “Look  at  the  task  lists  in  preparation  for  each  segment.” 

•  “Although  they  use  [a  different  format],  the  training  audience  found  it  useful  to  see 
another  product.” 

•  “Sample  products  are  good  for  format  and  how  it  meshes  with  the  order.” 

•  “Wish  [observers]  had  the  products  earlier,  to  use  in  preparation.” 
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Training  audience  ratings  of  the  brigade  staff  task  lists 
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Figure  15.  Feedback  from  the  training  audience  on  the  brigade  staff  task  lists. 


Adequacy  of  $taffin2  and  experience  specifications.  Because  of  the  importance  of 
limiting  the  number  of  support  personnel,  one  focus  of  the  evaluation  lay  in  scrutinizing  the  extent 
to  which  certain  participant  groups  were  adequately  staffed. 

•  Figure  16  presents  the  results  of  a  questionnaire  item  on  staffing  adequacy  for 

EXCON  and  blue  forces  roleplayers  and  observers.  In  general,  there  seemed  to  be  no 
strong  indication  that  staffing  levels  required  any  changes. 


Participant  ratings  of  the  extent  to  which 


support  personnel  staffing  levels  were  sufficient 


EXCON  Blue  Force  Observers  (n=9) 

Roleplayers  (pfS)  Roleplayers 

(n=19) 


Figure  16.  Feedback  on  the  adequacy  of  support  personnel  staffing. 


•  Another  question  was  addressed  to  the  simulation  center  OIC,  COBRAS  Coordinator, 
and  Blue  Forces  Controller,  asking  about  the  need  for  the  positions  of  Exercise 
Director,  COBRAS  Coordinator,  and  Blue  Forces  Controller.  Both  the  simulation 
center  OIC  and  the  COBRAS  Coordinator  responded  that  the  roles  were  essential. 

The  Blue  Forces  Controller,  however,  reported  that  his  position  was  not  necessary, 
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commenting  that  he  never  knew  what  exactly  he  was  supposed  to  do  during  the 
exercise.  This  may  reflect  the  continuous  presence  and  involvement  of  the  simulation 
site  staff,  who  monitored  the  blue  forces  workstations  themselves. 

Had  the  trial  been  implemented  in  strict  accordance  with  the  program  design,  the 
experience  levels  of  every  participant  would  have  closely  resembled  the  levels  recommended  in 
the  TSP.  Given  that  this  was  not  the  case,  it  was  judged  as  being  more  beneficial  to  ask 
participants  if  their  own  experience  levels  were  appropriate  for  the  roles  they  filled,  rather  than 
asking  participants  whether  or  not  the  intended  levels  would  have  been  correct.  The  results  are 
shown  in  Figure  17. 


Participant  responses  on  whether  they  had  the  right  amount  of  experience  for 

their  roles 


TF  Roleplayers  (n=9)  Other  BLUEFOR  EXCON  Roleplayers  Observers  (n=9) 
Roleplayers  (n=1 1 )  (n=6) 


Figure  17.  Feedback  on  the  experience  levels  required  for  Brigade  Staff  Exercise  participants. 

It  is  interesting  that  the  participants’  recommendations  concerning  experience  levels  were 
highly  consistent  with  the  recommendations  included  in  the  TSP.  Some  of  the  comments 
included: 

•  For  TF  roleplayers  -  experience  in  the  positions  roleplayed;  experience  in  tactical 
planning  and  orders  writing;  completed  the  Armor  Officer  Advanced  Course. 

•  For  other  subordinate  unit  roleplayers  -  present  or  former  staff  officer;  at  least  12 
months  experience  in  a  direct  support  platoon  in  support  of  a  brigade. 

•  For  EXCON  roleplayers  -  for  G3,  someone  who  has  worked  on  a  division  staff;  for 
division  aviation  operations,  attack  aviation  experience;  for  Division  Support 
Command,  experience  with  brigade  level  OPORDs  and  logistics;  for  division 
intelligence  officer,  battalion  or  brigade  intelligence  officer  experience;  for  division 
engineer,  minimum  combat  training  center  (CTC)  experience  as  assistant  brigade 
engineer;  for  division  fire  support,  division  level  experience. 

•  For  observers  -  must  have  been  in  the  job  or  branch  qualified  in  a  division  job. 

Pre-exercise  BBS  training  for  interactors  and  roleplayers.  The  pre-exercise  simulation 
training  for  interactors  and  roleplayers  was  also  of  interest  during  the  trial  implementation. 
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Developers  had  outlined  a  three-phase  training  plan  to  be  cx)nducted  by  the  simulation  site  staff, 
covering  BBS  fundamentals,  interactor  practice,  and  a  mini-exercise/BSE  rehearsal  for  interactors 
and  roleplayers.  However,  the  site  chose  to  conduct  their  own  train-up.  Therefore,  developers 
sought  information  regarding  how  the  BBS  training  could  be  made  most  adaptable  to  the  site 
staff,  while  still  achieving  the  COBRAS  goals,  as  listed  in  Section  4. 

The  findings  indicated  that  roleplayers,  interactors,  and  site  staff  all  felt  that  BBS 
familiarization  and  a  roleplayer/interactor  practice  session  were  both  necessary  and  sufficient. 
However,  it  was  also  observed  that  the  training  provided  by  the  simulation  site  staff  was,  in  this 
instance,  exceptionally  well  organized  and  implemented.  Therefore,  the  more  extensive  BBS 
training  plan  was  retained  in  the  TSP  in  the  hope  that  sites  with  less  well  defined  procedures 
would  use  it. 

Observation  and  feedback  procedures.  The  area  of  greatest  concern  on  the  observation 
and  feedback  system  dealt  with  the  frequency  of  AARs  and  their  perceived  benefit.  Observers 
were  asked  to  evaluate  the  AAR  structure.  Responding  to  questionnaire  items  addressing  the 
utility  of  fi’equent  AARs  and  the  extent  to  which  the  exercise  schedule  supported  sufficient  AAR 
preparation  time,  the  observers  (n=9)  were  supportive  of  the  plan  (see  Figure  18).  The  three 
questionnaire  items  included: 

•  To  what  extent  do  you  feel  that  frequent  AARs  (every  6-8  hours)  were  more  valuable 
than  a  single  summary  AAR  for  a  multi-day  exercise? 

•  To  what  extent  do  you  feel  that  the  AARs  were  valuable  for  the  brigade  staff? 

•  To  what  extent  was  there  enough  time  to  prepare  for  the  AARs? 


Figure  18.  Feedback  on  the  Brigade  Staff  Exercise  after  action  review  structure. 
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Several  observers  also  provided  comments  about  the  AAR  structure.  Those  comments 
included; 

•  I  felt  there  were  too  many  AARs,  which  prevented  me  from  doing  one-on-ones  with 
my  counterpart.  I  feel  there  should  be  three  AAR  times:  kind  of  a  beginning,  middle, 
and  end,  and  in-between  have  a  pause-ex  for  a  half-hour  and  have  the  0/Cs  give 
feedback  to  their  counterparts  and  then  see  if  they  do  some  adjustments  based  on  0/C 
recommendations. 

•  Time  was  a  little  tight,  but  [the  AARs]  are  pretty  informal,  and  “pre-done”  slides  in 
TSP  make  it  OK. 

•  The  one-on-one  AARs  between  the  observers  and  staff  are  most  productive  but  most 
difficult  to  fit  in  because  the  staff  is  so  busy. 

•  Frequent  AARs  are  good,  so  we  can  correct  as  we  go  along. 

•  AARs  as  “start-stop”  activities  seemed  to  disrupt  the  flow.  May  want  to  make  AARs 
more  on-demand,  when  they  have  trends. 

•  AARs  are  too  many  or  maybe  too  elaborate.  May  want  an  option  for  pausing  the 
exercise,  not  full  stop.  Might  be  better  than  pulling  up  stakes  for  AAR. 

•  Supplement  AARs  with  a  take-home  package. 

Exportabilitv.  A  key  focus  of  the  evaluation  was  on  measuring  the  extent  to  which  the 
trial  unit  could  implement  the  exercise  without  the  support  of  the  development  team-the  issue  of 
exportability.  Both  the  simulation  center  OIC  and  the  COBRAS  Coordinator  stated  that  they 
would  be  able  to  plan  a  BSE  implementation  for  the  first  time  by  relying  on  only  the  materials  in 
the  TSP,  without  the  support  of  the  developers.  In  addition,  they  both  strongly  agreed  that, 
having  run  the  exercise  once,  they  could  set  it  up  for  another  unit  or  coach  them  on  how  to  do  it. 

Implementation  conditions.  Another  design  question  that  was  investigated  concerned 
the  8-10  hours-per-day  training  schedule  used  in  the  BSE,  as  opposed  to  a  24-hour  continuous 
operations  arrangement.  Feedback  was  mixed  among  the  brigade  staff.  Some  indicated  that  the 
shorter  days  allowed  them  to  focus  on  the  training  issues  and  to  have  “time-out”  periods  in  which 
to  discuss  processes  or  catch  up.  Others  felt  that  the  24-hour  operations,  because  it  more  closely 
replicates  a  CTC  or  deployment  situation,  would  be  more  valuable.  Because  the  BSE  is  intended 
to  be  used  with  less  mature  staffs,  who  are  still  working  out  their  processes  and  require  less 
stressful  conditions,  these  responses  were  considered  to  be  supportive  of  the  8-10  hour  day. 
Figure  19  presents  the  distribution  of  training  audience  responses  on  training  day  length. 

The  final  design  issue  concerned  placement  of  CPs  in  the  simulation  center,  as  opposed  to 
putting  them  in  the  field.  Figure  20  indicates  the  training  audience  responses  to  the  question. 
Again,  their  responses  indicate  a  division  of  opinions.  The  same  reasoning  holds  for  this  issue  as 
for  the  previous  one:  that  the  decision  to  use  CPs  in  administrative,  indoor  settings  (and  less- 
than-24-hour-days)  was  made  in  order  to  support  optimal  learning  environments  for  the  staff, 
rather  than  to  present  complete  realism.  However,  one  observer  commented  that,  in  a  field  CP, 
the  staff  have  all  of  their  tools  and  know  where  they  are.  In  the  simulated  CPs,  they  had  to 
consciously  stock  and  locate  all  of  their  tools. 
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Training  audience  preference  for  8-10  hour  operations 
over  continuous  operations 
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Figure  19.  Feedback  concerning  8-10  hours-per-day  training. 


Training  audience  preference  for  locating  command  posts  In  the  Simulation 
Center  over  field  command  posts 
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Figure  20.  Feedback  on  location  of  command  posts. 
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Training  Support  Package  Findings 

The  second  component  of  the  trial  evaluation  focused  on  the  TSP.  The  evaluation 
approach  relied  on  questionnaires  that  addressed  all  aspects  of  the  TSP,  supplemented  by 
interviews  A\dth  participants  to  address  specific  areas  in  more  depth.  The  specific  issues  were; 

•  whether  the  TSP  provided  sufficient  description  of  various  aspects  of  the  BSE, 

•  the  extent  to  which  the  TSP  facilitated  performance  during  the  exercise,  and 

•  the  extent  to  which  the  TSP  was  organized  in  a  way  that  facilitated  its  use. 
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TSP  description  of  the  BSE.  One  function  of  the  TSP  is  to  describe  the  training  that  will 
occur,  including  how  it  should  be  conducted  and  how  the  TSP  materials  should  be  used.  One 
question  asked  how  well  the  TSP  described  the  exercise  in  general.  This  question  appeared  on 
every  questionnaire.  The  results,  as  shown  in  Figure  21,  were  mostly  positive  among  the 
roleplayers  and  observers,  while  training  audience  members  and  interactors  were  more  neutral  on 
the  question.  Most  of  the  administrative  participants  (simulation  site  OIC,  COBRAS 
Coordinator,  and  BBS  site  staff)  were  also  positive,  although  the  Blue  Forces  Controller  was  not. 


Participant  ratings  of  the  extent  to  which  their 
TSP  guides  described  the  exercise  accurately 
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■  Moderate  Extent 
□  Little  or  No  Extent 


Figure  21.  Feedback  on  the  description  of  the  exercise  in  the  participants’  guides. 


When  asked  whether  the  guides  helped  them  understand  their  roles,  most  responses  were 
again  positive  among  training  audience,  roleplayers,  and  observers  (see  Figure  22).  All  of  the 
administrative  participants  were  also  positive  about  the  descriptions  of  their  roles. 
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Participant  ratings  of  the  extent  to  which  their  ISP  guides 
helped  them  understand  their  roles 
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Figure  22.  Feedback  on  the  description  of  the  participant  roles  in  the  guides. 


TSP  facilitation  of  performance.  Another  aspect  of  the  TSP  that  was  evaluated 
addressed  the  effectiveness  of  the  TSP  in  facilitating  performance  during  the  exercise.  Although 
most  responses  from  the  training  audience  members,  the  brigade  staff  assistants,  and  the 
roleplayers  are  positive  (as  shown  in  Figure  23),  the  more  positive  responses  are  from  the  less- 
experienced  roleplayers,  with  more  temperate  responses  coming  from  experienced  staff  members. 


Participant  ratings  of  the  extent  to  which  their  TSP  guides  helped  them 
execute  their  roles 
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Figure  23.  Feedback  on  how  well  the  training  support  package  helped  participants  execute  their 
roles. 
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TSP  organization.  The  final  aspect  of  TSP  evaluation  addressed  the  extent  to  which  the 
guides  allowed  participants  to  find  information  in  the  TSP  as  they  prepared  for  the  exercise  and  as 
they  performed  during  the  exercise.  Overall,  as  shown  in  Figure  24,  the  design  of  the  TSP  was 
considered  adequate. 


Participant  ratings  of  the  extent  to  which  their  TSP  guides  facilitated  the 
referencing  of  infomnation 
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Figure  24.  Feedback  on  the  organization  of  the  training  support  package. 

Training  Benefit  Findings 

While  it  was  not  possible  to  directly  measure  the  effectiveness  of  the  training,  the 
questionnaires  did  address  perceptions  of  training  benefit.  Several  items  focused  on  the  general 
value  of  the  BSE.  The  items  were: 

•  To  what  extent  was  your  experience  worth  the  time  spent?  (asked  of  the  training 
audience,  brigade  staff  assistants,  roleplayers,  and  interactors) 

•  How  valuable  will  the  [BSE]  experience  be  to  you?  (asked  of  brigade  staff  assistants, 
roleplayers,  and  observers) 

•  How  does  the  BSE  compare  to  other  types  of  brigade  staff  training  in  terms  of  utility 
(benefit  versus  time  and  effort  required)?  (asked  of  the  training  audience) 

Responses  were  very  positive,  as  shown  in  Figure  25,  Figure  26,  and  Figure  27.  It  is 
interesting  to  note  that  participants  who  were  not  part  of  the  primary  training  audience  and  were 
classified  as  “supporters”  of  the  exercise  were  also  positive  about  the  value  of  the  exercise  for 
themselves.  This  generally  favorable  view  of  the  BSE,  as  indicated  by  participants  at  various 
levels,  is  supported  by  several  specific  questions  concerning  training  benefits. 
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Figure  25.  Feedback  on  whether  the  experience  was  worth  the  time  spent. 


Figure  26.  Feedback  on  the  Brigade  Staff  Exercise  value  for  non-training  audience  member 
participants. 


Figure  27.  Feedback  concerning  the  utility  of  the  BSE  and  other  brigade  staff  training. 
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To  obtain  impressions  of  how  the  BSE  benefited  participants,  participants  were  asked  to 
list  the  top  three  benefits  they  received  during  the  exercise.  Their  responses  varied  widely, 
indicating  a  great  diversity  of  perceived  benefits  (listed  in  Figure  28). 


Participant 

Group 

Benefit 

Training 

Audience 

Training  the  staffs  of  entire  brigade 
Rear  command  post  operations 
Information  flow 

Refine  SOP 

Reporting  process 

Tracking  battle 

Teamwork-first  time  staff  went 
through  this  process 

Section  personnel  training  in  DDMP 

Plan,  prepare,  and  execute  deliberate  using 

DDMP 

Orders  process/prep/brief 

Wargaming 

Mission  analysis  practice 

Learning  how  the  brigade  commander  fights 

Learning  brigade  commander’s  focus  for  my 

BOS 

Synchronization  and  integration  of  BOS 

Getting  familiar  with  current  brigade  staff 

Brigade  Staff 
Assistants 

Opportunity  to  work,  build 
teamwork  with  battle  staff 

Awareness  of  the  operational  plan 
Understanding  the  orders  process 
How  to  complete  reports 

Validation  of  SOP 

Developing  system  for  DDMP 

Planning  brigade  operations 

S2  planning-time  management 

Rehearsal 

Battle  tracking  and  synchronization 

Commander’s  expectations  and  views 

Identification  of  areas  to  improve 

Military  occupational  specialty-related  skills  for 
brigade/battalion  staff 

Roleplayers 

Task 

Force 

Improve  battalion  staff  DDMP 
Planning  process 

TOC  operations 

Synchronization 

Command  and  control 

Reporting 

Battle  tracking 

Utilize  new  charts  to  track  militaiy  intelligence 
assets 

Tracking  logistics 

Anticipating  logistical  needs  of  the  task  force 
Executing  resupply 

Validate  SOPs 

Work  with  other  staff 

Learn  how  orders  group  works 

Engineer 

Identify  areas  where  brigade  cannot 
support  engineer  logistics 

Procedure  for  requesting  supplies  from  bde 

Validate  engineer  battle  book 

Continued  on  next  page 

Figure  28.  Feedback  on  Brigade  Staff  Exercise  training  benefits. 
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Particrpsiiit 

Group 

Roleplayers 

Chemical 

Practice  briefing  my  unit’s 
capabilities  to  brigade 

Practice  digesting  brigade  order  and  figure  out 
how  it  applies  to  my  units 

Fire 

Radar  management 

Interaction  with  brigade  S2 

Support 

Ammo  management 

Experience  with  OPFOR  that  moved  and  fou^t 

Clearance  of  routes 

Executing  counterfire  missions  acquired  by  Q36 

FSB 

Learning  heavy  [organization] 
doctrine 

Better  understanding  of  duties  of  combat  health 
support  officer 

ADA 

Working  with  the  TOC 

EXCON 

Battle  tracking 

How  division  staff  operates 

OPFOR  tactics 

Synchronizing  division  fight 

Briefing  skills 

Interactors 

A  chance  to  play  OPFOR 

Introduction  to  possible  roles  in  the  field 

Able  to  see  the  big  picture 

Introduction  to  logistics  operations 

Working  with  other  cells 

Introduction  to  task  force  headquarters 

Working  with  the  TOC 

Making  the  TOC  do  its  job 
Understanding  battle  tactics 

Good  commo  exercise 

Saw  and  felt  the  reporting  systems 
Learned  more  about  brigade  SOP 

Overview  of  how  field  artillery 
battalion  works 

operations 

Learned  how  complex  combat  service  support 
operations  are 

Ensuring  all  slice  elements  have  good  cross-talk 
Exposure  to  division  decision-making 

Exposure  to  division  staff  and  coordination 
Dispersing  supplies,  evacuation,  maintenance 

Got  to  see  inter-branch  interactions 

Opportunity  to  experience  staff  operations  of 
combat 

Simulation  Site 

BBS  experience 

New  BBS  workaroimds 

Staff 

Game-building  experience 

A  look  at  other  databases  and  archives 

Figure  28  (continued).  Feedback  on  Brigade  Staff  Exercise  training  benefits. 


Other  items  asked  participants  to  estimate  the  extent  to  which  the  training  contributed  to 
their  knowledge  of  brigade  staff  battle  tasks,  gave  them  practice  opportunities,  or  helped  them 
improve  their  skills.  Responses  to  these  items,  again  generally  positive,  are  summarized  in  Figure 
29,  Figure  30,  and  Figure  31.  An  additional  question,  concerning  the  amount  of  practice  in  staff 
decision-making,  received  overwhelming  support  from  all  of  the  training  audience  members. 

It  is  gratifying  to  note  that  the  BSE  is  perceived  more  positively  in  terms  of  practice  and 
skill  acquisition  than  in  the  area  of  knowledge  acquisition,  as  the  BSE  is  designed  specifically  for 
tr^ning  and  practice,  rather  than  for  teaching  and  learning. 
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Participant  ratings  of  the  extent  to  which  the  BSE  increased 
their  knowledge  of  how  to  perform  battle  tasks 


Training  Audience 
{n=15) 


HGreatA/ery 
Great  Extent 

■  Moderate 
Extent 

□  Sight  or  No 
Extent 


Brigade  Staff  Assistants  Roleplayers  (n=1 9) 
{n=10) 


Figure  29.  Feedback  on  the  Brigade  Staff  Exercise  contribution  to  knowledge  of  how  to 
perform  battle  tasks. 


Participant  ratings  of  the  extent  to  which  the  BSE  provided  practice 
opportunities  on  battie  tasks 


Training  Audience  (n=1 5)  Brigade  Staff  Assistants  Rolepiayers  (n=1 9) 

(n=10) 


Figure  30.  Feedback  on  the  extent  of  Brigade  Staff  Exercise  practice  opportunities  on 
battle  tasks. 
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Participant  ratings  of  the  extent  to  which  they  were  able  to  improve 

battle  skills 


Training  Brigade  Staff  Roleplayers 

Audience  {n=1  ^  Assistants  (n=1 0)  (n=1 9) 


Figure  31.  Feedback  on  Brigade  StafFExercise  contribution  to  improvement  of  battle  skills. 

Finally,  participants  were  asked  about  the  use  of  BSE  within  a  brigade’s  overall  training 
strategy.  Specifically,  the  first  question  asked  about  the  value  of  the  BSE  for  brigade  staffs  with 
many  new  members;  all  of  the  training  audience  members  indicated  that  the  BSE  would  be  of 
great  value  to  newly-formed  brigade  staffs.  Another  question,  addressed  to  all  of  the  participants 
with  wider  involvement  in  brigade  training,  asked  what  priority  should  be  given  to  use  of  the  BSE 
for  such  brigade  staffs.  Among  the  training  audience  (n=15),  EXCON  roleplayers  (n=6),  and 
administrative  staff  (n=3),  96%  assigned  either  veiy  high  or  high  priority;  the  remaining  4%  (one 
person)  assigned  moderately  high  priority. 

A  third  item  asked  specifically  how  the  BSE  should  be  fit  into  a  brigade’s  overall  training 
strategy.  Responses  varied  fi'om  quarterly,  to  during  preparation  for  field  exercises,  to  being  used 
as  evaluation  event  to  determine  staff  improvement.  Their  responses  are  shown  in  Figure  32. 

Program  Revisions 

All  in  all,  the  evaluation  results  from  the  trial  implementation  were  encouraging.  The  BSE 
design  decisions  were  supported,  the  TSP  was  perceived  as  adequate  in  content  and  structure,  and 
the  BSE  is  seen  as  providing  training  benefit,  especially  for  newly-formed  brigade  staffs.  Various 
minor  errors  were  caught  (or  pointed  out  by  participants),  all  of  which  were  noted  and  corrected. 
But  no  serious  design  or  content  flaws  were  discovered. 

There  were  many  indicators  for  future  development,  however.  While  the  BSE  was 
observed  to  be  a  useful  training  tool  for  brigade  staffs  (especially  less  mature  staffs)  that 
emphasized  the  decision-making  process  and  CSS  activities,  other  areas  of  concern  in  brigade 
operations  were  discussed.  Some  of  these  areas  are  discussed  in  Section  7. 
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Training 


Comments  on  Use  of  BSE  in  Brigade  Training  Strate^ 


Participants 


Training  Audience  Everywhere.  I’d  conduct  it  a  couple  of  times  a  quarter.  Given  the  orders  and 

supporting  documentation,  even  without  BBS,  great  value  to  working  the 
orders  process  only. 

Excellent  training  package  to  train  a  new  battle  staff,  plus  it  provides  an  excellent 
low  cost,  low  overhead  vehicle  to  conduct  staff  sustairunent  training. 

Should  be  done  quarterly  to  keep  warfighting  skills  sharp. 

Has  to  happen  early,  often,  and  as  a  graduation  exercise 

I  think  it  needs  to  be  a  quarterly  event.  The  problem  would  be  in  having 
[observers]  eveiy  quarter. 

Excellent  initial  and  mid-point  training  event. 

A  train-up  for  future  field  exercises. 

Initially,  it  is  a  great  tool  to  see  what  level  the  brigade  staff  is  at.  After  several 
exercises  (in  a  field  environment),  it  would  be  a  great  tool  to  see  how  much  the 
staff  has  improved. 

Administrative  Staff  As  a  validation  gate  or  checkpoint  prior  to  a  major  field  training  exercise  or  CTC 

rotation. 

Excellent  tool  as  a  team-builder  for  new  staff.  Therefore,  exercise  should  take 
place  after  significant  staff  turnover.  Should  also  be  used  as  a  pre-cursor  to  a 
_ NTC  train-up  and  also  as  a  validation  exercise  just  prior  to  the  CTC  rotation. 


Figure  32.  Feedback  on  how  the  Brigade  Staff  Exercise  should  be  included  in  a  brigade’s  training 
strategy. 


Summary 

The  formative  evaluation  and  trial  implementation  of  the  BSE  were  parts  of  an  ongoing 
large-scale  effort  that  required  considerable  time  and  effort.  However,  they  were  necessary  in 
order  to  obtain  the  information  from  both  users  and  observers  concerning  the  scenario,  the  TSP 
materials,  and  the  implementation  model. 

Although  the  vignettes  were  visibly  different  from  the  BSE,  they  were  built  on  much  of  the 
same  foundation  as  the  BSE.  The  clearest  linkage  is  in  the  training  audience,  the  scenario  events, 
and  the  performance  objectives.  They  are  also  similar  with  respect  to  the  design  and  development 
cognitive  processes  and  TSP  structure,  even  though  the  actual  TSP  materials  have  a  very  different 
appearance.  The  vignette  development  and  evaluation  are  described  in  the  following  section. 
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SECTION  5;  DEVELOPMENT  AND  PILOT  TESTING  OF  THE 
BRIGADE  STAFF  VIGNETTES 

Unlike  the  BSE,  the  COBRAS  n  vignettes  were  to  supplement,  rather  than  replace  the 
COBRAS  I  work.  In  COBRAS  1, 13  vignettes  had  been  developed,  although  two  were  pulled 
from  production  because  they  were  simply  not  ready  for  use.  In  COBRAS  H,  1 1  new  topics  or 
events  were  selected  for  vignette  development.  Additionally,  the  two  vignettes  that  were  still 
incomplete  from  COBRAS  I  were  to  be  reengineered  and  finalized. 

The  original  1 1  vignettes  had  still  not  been  implemented  or  tested  at  the  time  of  COBRAS 
n  development.  Thus,  strictly  speaking,  there  were  no  trial  results  or  “lessons  learned”  to  guide 
the  COBRAS  n  development.  However,  instructional  design  principles  were  applied  again  to  the 
model,  resulting  in  minor  changes  in  the  way  that  the  vignettes  were  structured  and  presented. 

The  refined  model  has  been  fiilly  documented  in  C.  H.  Campbell,  Ford,  and  Campbell  (in 
preparation).  This  section  of  the  report  highlights  the  process  of  development.  The  topics 
covered  include: 

•  Initial  Vignette  Development:  The  SOW  (ARJ,  1995)  required  that  12  additional 
\agnettes  be  developed,  focusing  on  the  newly  expanded  target  training  audience.  At 
least  two  of  the  vignettes  were  also  to  incorporate  vertical  linkages  with  the  brigade’s 
maneuver  battalions.  The  use  of  virtual  or  constructive  simulation  was  to  be  explored, 
and  the  same  scenario  that  was  used  in  the  BSE  was  to  underlie  the  vignettes.  This 
section  describes  how  developers  worked  from  that  guidance  to  designate  the  specific 
new  audience  members. 

•  Conduct  of  the  Pilot  Test,  Results,  and  Revisions:  The  pilot  test  design  and 
implementation  are  described,  and  results  and  subsequent  revisions  are  summarized. 

•  Final  Internal  Review:  In  the  absence  of  a  trial  implementation,  an  intense  and 
rigorous  internal  review  was  conducted,  resulting  in  significant  changes  to  the 
vignettes.  The  final  package  and  TSP  materials  are  described. 

Initial  Vignette  Development 

Development  of  the  vignettes  began  shortly  after  the  initial  BSE  decision-making  was 
completed.  As  described  in  Section  3,  COBRAS  staff  had  designated  the  16-person  training 
audience,  performed  preliminary  design  work  on  the  scenario  for  the  three  missions,  and  started 
the  ModSPA  analysis  of  job  requirements  (tasks).  These  three  components  of  the  initial  design 
work-training  audience,  scenario,  and  tasks-formed  the  foundation  for  vignette  development. 

The  vignette-specific  training  audience  groups  were  subsets  of  the  full  BSE  training  audience. 
Likewise,  the  vignette-specific  events  were  derived  from  the  overall  BSE  scenario,  although  some 
METT-TC  elements  were  allowed  to  vary  in  order  to  focus  the  vignettes  on  specific  activities. 

The  ModSPA  and  the  resulting  task  lists  and  sample  products  served  as  the  basis  for  the 
vignette  objectives.  Vignette  developers  used  the  basic  individual  task  lists  to  derive 
corresponding  small  group  performance  lists  addressing  the  event  and  activities  of  the  vignette. 
These  process  lists  were  fijrther  expanded  into  a  series  of  probing  AAR  questions  that  the 
vignette  training  leader  uses  to  stimulate  discussion  of  the  exercise. 
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Following  the  same  process  of  scenario  examination  used  in  COBRAS  I,  developers 
prepared  drafts  of  12  vignettes.  For  each  vignette,  internal  expert  reviews  of  tactical  content, 
instructional  value,  and  performance  description  were  conducted.  Each  was  enacted  several  times 
within  the  project  staff  to  check  on  quality  and  consistency.  In  the  process,  one  vignette  was 
dropped  because  it  was  too  broad,  both  in  content  and  in  audience,  to  fit  the  definition  of  a 
vignette. 

At  the  same  time,  the  approach  to  simulation-based  vignettes  was  receiving  another  hard 
examination.  In  terms  of  benefits,  simulation  has  the  potential  for  providing  significant  training 
benefit.  There  are  at  least  six  ways  in  which  simulation  could  enhance  training,  including: 

1.  EflBciently  replicate  battlefield  activity,  using  simulated  units  to  stand  in  for  large 
support  requirements. 

2.  Generate  cues  for  training  audience  task  performance  as  opposed  to  trying  to  manually 
generate  dynamic  conditions  of  battlefield. 

3.  Provide  outcomes  to  battlefield  actions  which  are  reasonable. 

4.  Provide  standardized  training  conditions. 

5.  Provide  instructional  features  that  enhance  efficiency  and  effectiveness,  including 
performance  measurement. 

6.  With  replay  capabilities,  can  support  providing  objective  feedback. 

However,  if  the  simulation  does  not  significantly  achieve  one  or  more  of  these  advantages, 
then  it  is  not  an  enhancement  to  training.  In  the  case  of  vignettes,  COBRAS  developers  found 
that  simulations  were  often  not  of  sufficient  benefit  to  justify  the  resource  cost.  This  is  primarily 
because  staffs  “see  the  battle”  fi'om  their  CPs  via  reports,  messages,  and  other  means.  If  the 
events  being  trained  are  of  relatively  short  duration  or  are  not  particularly  dynamic,  a  simulation  is 
often  not  necessary.  It  is  relatively  easy  to  produce  scripted  cues  or  inputs  sufficient  to  drive  the 
training  of  these  types  of  events.  In  fact,  the  use  of  a  simulation  will  likely  drive  up  support  costs 
in  these  cases.  Additionally,  the  short  time  duration  of  vignettes  is  usually  insufficient  to  allow  tlie 
simulation  to  fully  portray  outcomes  of  decisions  and  actions.  And  for  vignettes  that  focus  on 
group  interactions  and  synchronization,  instructional  features  such  as  replay  capabilities  are  rarely 
useful  (although  they  may  be  appropriate  for  evaluating  tactical  decision-making). 

Thus  the  cost-benefit  considerations  lead  to  the  conclusion  that  simulation  is  a  significant 
contributor  to  training  value  only  if; 

•  the  vignette  scenario  requires  representation  of  weapon  systems  or  troops  to  provide 
essential  tactical  realism,  dynamically  cue  the  training  audience,  and  react  to  the 
training  audience;  and 

•  the  vignette  can  be  extended  to  more  than  four  hours,  to  allow  the  situation  to 
develop. 

The  alternative  to  using  technology-based  simulations  is  to  use  “live”  simulation.  In  the 
context  of  most  vignettes,  this  is  relatively  easily  accomplished.  The  training  participants, 
normally  located  in  a  CP,  can  perform  the  vignette  in  an  administrative  CP  (i.e.,  any  room  large 
enough  to  work  in).  Communications  may  be  represented  by  means  of  written  messages  or  by 
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having  the  training  monitor  simply  provide  any  necessary  information.  By  using  this  type  of 
simulated  environment,  resource  costs  are  kept  low,  and  the  vignettes  become  more  feasible  as 
unit-administered  exercises. 

For  these  reasons  most  of  the  vignettes  developed  for  COBRAS  11  were  designed  for 
“live”  simulation.  As  in  COBRAS  I,  the  staff  explored  use  of  simulation  in  two  vignettes.  Based 
on  lessons  learned  in  the  BSE  trial  implementation,  the  materials  to  support  the  simulation 
operation  were  revised  for  the  two  COBRAS  I  simulation-based  vignettes  as  well. 

During  the  COBRAS  n  vignette  development  process,  12  new  vignettes  were  developed, 
as  well  as  the  two  substantially  re\ased  COBRAS  I  vignettes.  The  evaluation  plan  called  for  trial 
implementations  of  all  of  the  vignettes  using  brigade  personnel  in  all  roles.  This  plan  required 
assistance  from  an  actual  brigade  staff  for  a  minimum  of  10  days,  as  well  as  participation  of 
members  of  maneuver  battalion  staffs  for  four  days.  While  developers  were  aware  that  the 
request  would  prove  difficult  to  fill,  it  remained  the  only  way  to  fully  validate  the  vignette  designs 
and  TSP  materials. 


Conduct  of  the  Pilot  Implementation,  Results,  and  Revisions 


Because  the  actual  assets  required  for  the  planned  trial  implementation  were  unavailable,  it 
was  necessary  to  revise  the  evaluation  plan.  A  second  plan  was  prepared,  comprising  the  pilot 
implementation  of  a  limited  set  of  vignettes  with  roleplayed  brigade  and  battalion  staff  members. 


The  Pilot  Implementation  Plan 

Five  vignettes  were  selected  for  the  pilot  implementation.  They  were  selected  as  being 
representative  of  the  complete  set  in  terms  of  their  lengths  and  number  of  participants,  as  well  as 
being  the  vignettes  for  which  appropriately  experienced  participants  were  available.  The  five  pilot 
vignettes  included: 


Vignette  14: 
Vignette  16: 
Vignette  19: 
Vignette  22: 
Vignette  23: 


Develop  a  Reconnaissance  Order 

Plan  Deliberate  Smoke  Operations 

Conduct  Abbreviated  Decision-Making  Process 

Conduct  a  Brigade  Rehearsal 

Plan  a  Combat  Service  Support  Rehearsal 


The  participants,  while  not  members  of  intact  staffs,  would  have  the  equivalent  levels  of 
staff  experience  as  the  target  training  audience  members.  The  feedback  obtained  from 
observations  and  from  the  participant  interviews  would  be  interpreted  with  caution,  in  light  of  the 
disparity  between  the  constitution  of  the  pilot  participant  group  and  the  targeted  training  group. 
Information  would  be  generalized  to  the  full  set  of  COBRAS  II  vignettes  to  assist  in  refining  the 
materials  and  design. 


Pilot  participants  included  groups  of  U.S.  Army  personnel  from  Fort  Knox  (armor, 
engineer,  and  chemical  experts),  Fort  Lee  (logistics  experts).  Fort  Sill  (field  artillery  experts),  and 
Fort  Huachuca  (intelligence  experts).  One  individual  from  a  contract  effort  at  Fort  Knox  served 
as  the  Training  Coordinator.  Normally,  it  is  recommended  that  the  Training  Coordinator,  who 
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facilitates  the  wgnette  preparation,  conduct,  and  discussion,  be  the  unit  XO.  The  Training 
Coordinator  roleplayer  had  recent  experience  as  a  training  observer,  but  not  as  a  brigade  XO. 

Pilot  Implementation  Findings 

Several  key  points  emerged  from  the  pilot  implementation  of  the  vignettes.  These  points 
were  synthesized  from  developer  observations  and  participant  comments.  In  summary,  they 
included; 

•  Participants  were  often  confused  regarding  precisely  what  they  were  to  do  during  the 
vignette. 

•  A  lack  of  specificity  in  the  tactical  products  (e  g.,  commander’s  guidance)  frequently 
led  to  “dead  ends,”  in  which  participants  and  the  Training  Coordinator  were  unable  to 
determine  the  significance  or  even  the  meaning  of  information  and  could  not  proceed. 

•  Training  objectives  and  tasks  were  not  always  well-defined,  and  did  not,  in  those 
cases,  support  the  vignettes  in  providing  learning  for  participants. 

•  The  designated  participants  for  some  vignettes  were  not  appropriate  for  the  vignette 
activities;  in  particular,  materials  sometimes  required  participation  for  only  small, 
insignificant  activities  within  the  scope  of  the  vignette. 

•  Excessive  amounts  of  tactical  materials  were  provided  in  the  TSPs,  which  proved  to 
be  overwhelming  as  participants  tried  to  absorb  the  tactical  situation. 

The  active  involvement  of  COBRAS  vignette  developers  enabled  the  participants  to  work 
through  most  of  the  difficulties,  and  a  great  deal  of  valuable  and  specific  information  that  was 
helpful  in  refining  the  vignettes  was  obtained.  However,  the  overarching  conclusion  was  that  the 
vignette  TSPs  left  participants  confused  as  to  what  they  were  trying  to  accomplish  during  the 
vignettes. 


Final  Internal  Review 

After  the  pilot  testing,  the  notes  and  observations  were  aggregated  and  analyzed  in  order 
to  determine  changes  necessary.  Because  only  five  of  the  vignettes  were  tested,  however,  the 
COBRAS  staff  attempted  to  generalize  the  results  to  all  of  the  vignettes.  In  the  process,  it 
became  obvious  that  all  of  the  vignettes  suffered  from  internal  inconsistency  in  the  tactical 
situations  and  materials,  lack  of  clarity  in  the  instructions  and  guidance,  and  vague  or  even 
indefensible  statements  of  performance  requirements. 

The  conclusions  were  daunting  but  inescapable:  The  vignettes  required  significant  review 
and  revision  on  all  aspects,  including  tactical  content,  instructional  design,  and  performance 
requirement  specifications.  In  the  absence  of  U.S.  Army  military  personnel  on  whom  to  try  out 
the  vignette  materials,  rigorous  evaluations  would  be  conducted  using  COBRAS  personnel  and 
staff  from  related  projects. 

The  review  process  took  the  form  of  a  complete  reconstruction  of  all  of  the  vignettes.  A 
series  of  internal  workshop  sessions  was  instituted  to  guide  developers  through  the  design  and 
development  work.  Developers  worked  in  teams  of  SMEs  and  training  developers,  guided  by 
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four  senior  training  developers  on  the  COBRAS  staff.  The  workshop  sessions  led  the  teams  back 
through  the  full  development  process. 

As  an  outcome  of  the  workshop,  all  of  the  non-simulation  vignettes  were  overhauled.  It 
required  additional  guidance  and  time  to  complete  the  simulation-based  vignettes  (both  the  two 
new  ones  and  the  two  COBRAS  I  vignettes  that  were  being  reconstructed).  Thorough  reviews 
and  in-house  trials  of  each  vignette  were  useful  in  discovering  and  correcting  inconsistencies  in 
tactical  materials.  The  final  complete  set  of  live  and  constructive  simulation-based  vignettes,  from 
both  COBRAS  I  and  COBRAS  H,  is  shown  in  Figure  33. 

A  top-down  review  of  the  vignette  TSP  structure  also  led  to  a  redesigned  structure  for  the 
materials  (see  Figure  34).  Finally,  the  Guide  to  Use  and  Implementation  of  Vignettes  was 
modified  to  include  the  new  vignettes  and  to  correct  guidance  concerning  the  simulation-based 
vignettes. 

Another  outcome  of  the  workshop  was  a  formal  documentation  of  a  methodology  for 
development  of  vignettes  (C.  H.  Campbell,  Ford,  &  Campbell,  in  preparation).  The  methodology 
is  based  on  the  more  general  Methodology  for  Development  of  Structured  Simulation-Based 
Training  {C.  H.  Campbell,  Deter,  &  Quinkert,  1997),  with  specific  guidance  pertaining  to  small 
group  exercises.  That  specific  guidance  concerns  the  detailed  event  analysis  that  must  precede 
development,  the  requirement  to  trim  the  audience  to  essential  participants  and  to  provide  only 
the  necessary  and  sufficient  tactical  materials,  and  considerations  in  formulating  AAR  questions. 
The  methodology  is  summarized  in  Appendix  C. 

Summary 

In  retrospect,  it  is  clear  that  carving  a  small  segment  of  staff  performance  out  of  the 
context  in  which  it  is  usually  nested  is  very  difficult.  Deciding  who  is  involved  in  the  activity,  as 
opposed  to  who  might  be  involved,  requires  hard  choices  that  are  always  subject  to  second 
guessing.  Creating  efficient  products  and  processes  that  lead  up  to  the  event  requires  creativity 
and  a  detailed  knowledge  of  what  is  going  on  in  staff  processes  and  in  the  minds  of  staff  members. 
Sorting  “nice  to  know”  from  “must  have”  information  requires  dissecting  how  decisions  are  made 
and  what  they  are  based  on.  Finally,  creating  products  that  are  logical  end  points  or  outputs  of 
the  events  requires  detailed  knowledge  of  formal  and  informal  staff  products  which  reflect 
matched  activities  and  bound  just  the  event  selected  for  training. 

All  of  these  factors  make  complete,  efficient  vignettes  one  of  the  most  challenging  training 
development  efforts.  As  developers  finally  produced  the  vignette  TSPs,  it  became  apparent  that 
development  of  a  single  small  group  exercise  is  not  much  easier  than  development  of  a  full 
integrated  exercise,  such  as  the  BSE.  The  lack  of  full  trial  implementations  was  unfortunate, 
although  the  limited  pilot  tests  yielded  crucial  information  concerning  the  overall  set  of  vignettes. 
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1 

Plan  for  Dislocated  Civilians 

S1,S2,  S4 

2 

Plan  Refuel  on  the  Move 

S4,  FSB  Cdr 

3 

Develop  a  Concept  of  Service  Support 

S1,S4 

4 

Develop  a  Reconnaissance  and  Surveillance  Plan 

S2,  S3 

5 

Conduct  Target  Development 

XO,  S2,  S3,  FSO 

6 

Develop  Air  Defense  Concept 

S2,  S3,  ADCOORD 

7 

Develop  Contingency  Plan 

S2,  S3,  FSO,  ENG 

8 

Conduct  Mission  Analysis 

XO,  SI,  S2,  S3,  S4,  FSO,  ENG,  ADCOORD 

9 

Develop  Courses  of  Action 

XO,  SI,  S2,  S3,  S4,  FSO,  ENG,  ADCOORD 

10 

Conduct  Course  of  Action  Analysis 

XO,  SI,  S2,  S3,  S4,  FSO,  ENG,  ADCOORD 

11 

Conduct  Special  Staff  Rehearsal 

XO,  S2,  S3,  FSO,  ENG,  ADCOORD 

12 

Develop  a  Reconnaissance  Order 

S2,  S3,  S4,  FSO,  ADCOORD,  ENG,  SIGO,  MI 
Co  Cdr,  CHEMO 

13 

Develop  a  Course  of  Action  Branch 

S3,  FSO,  AVN  LNO,  ENG 

14 

Plan  Nuclear,  Biological,  and  Chemical  Defense 
Operations 

S2,  S3,  CHEMO 

15 

Plan  Deliberate  Smoke  Operations 

S2,  S3,  FSO,  CHEMO 

16 

Plan  Brigade  Rear  Battle 

S2,  S3,  FSO 

17 

Plan  Combat  Service  Support  Rehearsal 

S1,S4,  FSB  Cdr 

18 

Identify  and  Resolve  Airspace  Conflicts 

S3,  S3-Air,  FSO,  AVN  LNO,  ALO, 

ADCOORD 

19 

Conduct  a  Brigade  Rehearsal 

Bde  Cdr,  XO,  S2,  S3,  S4,  FSO,  FSCOORD, 

ENG,  ADCOORD,  CHEMO,  Bn/TF  Cdrs 

20 

Conduct  Accelerated  Decision-Making  Process 

Bde  Cdr,  XO,  SI,  S2,  S3,  S4,  FSO,  FSCOORD, 
ENG,  ADCOORD,  CHEMO,  SIGO,  MI  Co  Cdr 

21 

Coordinate  Mission  Operations 

XO,  S2,  S3,  FSO,  ENG,  ADCOORD  (Janus) 

22 

Coordinate  a  Mission  Transition — Offense  to 

XO,  SI,  S2,  S3,  S4,  FSO,  ENG,  ADCOORD, 

Defense 

FSB  Cdr  (BBS) 

23 

Conduct  Parallel  Planning 

Bde  Cdr  XO,  SI,  S2,  S3,  S4,  FSO,  ENG, 
ADCOORD,  FSB  Cdr,  CHEMO,  MI  Co  Cdr 
(BBS) 

24 

Plan  and  Execute  a  Fragmentary  Order 

Bde  Cdr,  XO,  S2,  S3,  FSO,  FSCOORD,  ENG, 
ADCOORD,  CHEMO  (Janus) 

Figure  33.  Titles  and  participants  of  the  COBRAS  II  vignettes. 


55 


TRAINING  COORDINATOR  AND  PARTICIPANT  GUIDES 

•  BACKGROUND  -  Familiarize  Training  Coordinator  and  participants  with  basic  requirements 
for  the  vignette-participants,  time,  training  coordinator  duties. 

•  SCOPE  -  Describes  vignette  in  terms  of  activities,  training  objective,  and  limitations  on 
activities. 

•  PREPARATION  -  How  Training  Coordinator  prepares  self  and  participants;  how  participants 
prepare.  List  of  preparation  materials.  Copy  of  administrative  brief  used  for  initial  vignette 
orientation  for  participants. 

•  EXECUTION  -  How  to  start  and  conduct  this  vignette,  list  of  execution  materials,  situation 
brief  used  to  initiate  the  vignette  (for  Training  Coordinator). 

•  After  Action  Review  (AAR)  -  How  to  facilitate  AAR  and  use  sample  products,  list  of  AAR 
questions  and  response  considerations  (for  Training  Coordinator). 

•  REFERENCES 

TACTICAL  MATERIALS  FOR  PREPARATION 
TACTICAL  MATERIALS  FOR  EXECUTION 
TACTICAL  AND  PERFORMANCE:  JOB  AIDS 
SAMPLE  VIGNETTE  PRODUCTS 
SUPPORT  MATERIALS  (for  simulation-based  vignettes) 

Figure  34.  Structure  and  contents  of  the  COBRAS  II  vignettes. 

The  following  two  sections  summarize  lessons  learned  from  both  BSE  and  vignette 
development.  Section  6  focuses  on  lessons  learned  for  future  training  development  projects  that 
are  along  the  same  lines  as  COBRAS  I  and  II.  Section  7  draws  together  many  of  the  suggestions 
and  indications  of  training  needs  that  may  guide  initiation  of  future  development  projects. 
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SECTION  6:  LESSONS  LEARNED  FOR  THE  DEVELOPMENT  OF  FUTURE 

TRAINING 

Over  the  past  8  years,  ARI  and  its  contractor  teams  have  accomplished  a  number  of  R&D 
efforts  to  incorporate  structure  into  battalion  and  brigade-level  training  programs.  The 
preliminary  work  in  development  of  structured  training  scenarios  was  focused  on  the  definition  of 
threat  conditions  (R.  C.  Campbell  &  Campbell,  1990;  Doyle,  1990).  This  work  was  vastly 
expanded  in  the  initial  development  and  expansions  to  the  VTP  (Hoffman  et  al.,  1995;  Graves  & 
Myers,  1997;  Koger  et  al.,  1996)  and  the  development  of  the  COBRAS  I  program  (Graves  et  al., 
1997).  Initial  evaluations  of  such  programs  have  indicated  that,  Avith  a  well-developed  training 
structure,  simulation  can  contribute  substantially  to  the  value  of  training  (Shlechter,  Bessemer, 
Nesselroade,  &  Anthony,  1995). 

Program  developers  and  sponsors,  however,  are  well  aware  that  the  structured  approach 
and  development  methodologies  can  always  be  improved.  To  that  end,  each  of  the  ARI  projects 
has  produced  a  report,  documenting  “lessons  learned”  that  may  be  useful  in  future  development 
efforts.  The  lessons  learned  during  the  COBRAS  n  project  represent  the  latest  set,  and  are 
described  below. 

This  project's  lessons,  which  are  few  in  number  but  of  broad  scope,  were  drawn  from 
formative  evaluation  efforts*  in  which  design  solutions  and  training  support  materials  were 
evaluated  in  terms  of  their  quality  and  their  support  of  explicitly  defined  training  objectives.  The 
sk  lessons  described  below  focus  on; 

1.  the  development  process, 

2.  program  design  characteristics, 

3.  program  implementation  requirements, 

4.  use  of  training  support  products, 

5.  the  nature  of  lessons  learned,  and 

6.  the  value  of  structured  training  and  TSPs. 

All  of  these  lessons  should  be  of  use  to  future  program  developers,  whether  they  are 
design  scientists  who  are  responsible  for  delineating  project  objectives  and  design  alternatives,  or 
military  and  instructional  specialists  who  construct  TSPs. 

Development  Process 

The  development  methodology  (C.  H.  Campbell  et  al.,  1995),  described  in  Appendix  C,  is 
the  model  by  which  the  COBRAS  II  exercises  were  developed.  During  the  course  of  the  project, 
developers  were  able  to  validate  lessons  learned  during  COBRAS  I:  that  the  initial  decisions  and 
constraints  must  be  identified  as  completely  as  possible,  and  that  the  conduct  of  all  formative 
evaluation  activities  is  critical. 


®  Here  again,  as  throughout  this  report,  “formative  evaluation”  refers  to  the  project-long  cycle  of  examine-evaluate- 
refine  (as  described  in  Section  2). 
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But  developing  training  such  as  this  is  not  an  intuitive  process.  Being  experienced  in 
training  or  having  done  a  great  deal  of  training  doesn’t  translate  directly  to  being  able  to  develop 
good  TSPs.  There  is  a  global  lesson  concerning  the  use  of  a  methodology  in  training 
development: 

Using  a  procedural  methodology  during  development  of  training  is  critical 
in  producing  a  high-quality  program. 

Developmental  discipline  and  having  a  sound  process  and  procedure  to  follow  will  help  to 
preclude  omissions  and  “reinventing  the  wheel.”  The  methodology  for  development  of  structured 
simulation-based  training  (C.  H.  Campbell  et  al.,  1995)  was  understood  by  all  of  the  COBRAS 
developers  before  the  COBRAS  II  development  began.  Because  it  incorporated  many  of  the 
lessons  learned  and  experience  from  COBRAS  I  and  previous  projects,  it  spoke  directly  to  the 
activities  required  in  COBRAS  H.  As  a  result,  the  development  and  evaluation  process  for  the 
BSE  was  relatively  straightforward. 

For  the  vignettes,  on  the  other  hand,  no  methodology  had  yet  been  articulated  or 
formalized.  During  development,  there  was  always  a  sense  that  something  was  missing,  that 
certain  steps  were  not  understood,  and  that  essential  checks  and  reviews  were  not  being  done. 
The  pilot  implementation  verified  our  fears:  the  products  were  not  sufficiently  clear,  correct,  or 
well-organized.  Formalization  of  the  process  during  the  vignette  development  workshop  brought 
all  developers  on  line  with  respect  to  the  steps  in  development  and  the  standards  for  products. 

Use  of  the  vignette  methodology  in  future  small  group  exercise  efforts,  or  of  similarly  well 
thought  out  methodologies  for  other  emphases,  can  only  strengthen  the  quality  and  utility  of  the 
efforts. 


Program  Design  Characteristics 

The  second  lesson  is  as  follows: 

Structured  training  programs  should  be  flexible  enough  to  allow  for 
implementation  in  various  settings. 

However,  this  lesson  does  not  stand  alone: 

Corollary:  The  more  changes  you  make,  the  more  you  need  to  understand 
the  details  of  the  exercise  that  you  're  changing. 

The  lesson  itself  is  drawn  from  observations  and  comments  during  the  December  trial 
implementation  of  the  BSE,  as  well  as  from  the  earlier  trial  of  the  COBRAS  I  exercise  at  Fort 
Riley.  Although  it  recommends  increased  flexibility,  it  does  not  minimize  the  structured  nature  of 
the  training.  The  focus  on  performance  of  specific  stated  training  objectives  for  selected  members 
of  a  unit  remains  as  the  primary  objective  and  goal. 

However,  certain  characteristics  of  the  implementation  required  adjustment  and 
modification  during  the  trial  implementations.  One  was  the  AAR  schedule.  The  exercise  design 
called  for  formal,  slide-supported  AARs  at  specified  points  in  the  exercise.  In  both 
implementations,  the  schedule  was  modified  within  the  first  day  of  the  exercise  start  in  order  to 
provide  more  performance  time  as  well  as  more  AAR  preparation  time.  In  both  instances,  one 
AAR  was  dropped  in  the  interests  of  time. 
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Fortunately,  by  the  time  of  the  December  trial  implementation,  developers  had  anticipated 
the  need  for  schedule  adjustment.  The  TSP  contained  instructions  for  schedule  modification,  as 
well  as  guidance  on  the  purpose  of  the  AARs  and  how  they  tied  together  across  the  exercise.  As 
a  result,  the  adjustment  was  made  and  the  critical  feedback  was  provided  to  the  unit.  Several 
observers  made  time  for  informal  group  AARs  in  order  to  provide  coaching  and  feedback  without 
the  benefit  of  slides  or  other  more  disruptive  means. 

The  exercise  was  also  modified  in  terms  of  task  organization  of  the  participating  brigade, 
training  audience  participation,  and  use  of  AAR  materials.  COBRAS  developers  were  consulted 
on  the  effects  of  changes  to  the  task  organization,  and  the  questions  and  answers  were 
incorporated  into  the  TSP  following  the  trial  implementation. 

In  two  other  areas,  brigade  member  participation  and  AAR  structure,  brigade  members 
and  observers  simply  made  the  changes  that  they  felt  were  necessary.  The  changes  were  made 
easily,  the  training  was  not  compromised,  and  participants  expressed  gratification  that  the 
flexibility  was  permitted. 

The  corollary  to  the  lesson  derives  from  a  third  implementation  of  the  COBRAS  11  BSE 
that  took  place  at  Fort  Lewis  in  May  1997.  This  implementation  involved  significant  alterations 
to  the  BSE  model,  in  task  organization  (use  of  one  armor,  one  mechanized  infantry,  and  one  light 
infantry  battalion),  schedule  (overlapping  missions  and  concurrent  planning  and  operations,  rather 
than  sequential  missions),  and  OPFOR  activity  and  aggressiveness.  Because  of  the  differences, 
this  implementation  was  not  considered  to  be  an  appropriate  opportunity  for  formative  evaluation 
data  collection.  However,  COBRAS  developers  did  assist  with  and  observe  the  exercise. 

As  a  result  of  the  locally-made  modifications  to  provide  for  overlapping  missions,  there 
was  considerable  confusion  about  the  intelligence  message  traffic:  The  messages  had  been 
carefully  timed  to  support  a  scenario  that  would  cue  brigade  performance,  and  disrupting  the 
timing  not  only  made  the  messages  confusing  or  incorrect,  but  also  caused  certain  brigade  tasks  to 
be  not  cued.  The  brigade  tasks  and  AARs  that  were  tied  to  the  scenario  were  no  longer  correct: 
Observers  were  forced  to  figure  out  what  tasks  to  observe  and  what  important  behaviors  of  the 
brigade  staff  should  be  the  subject  of  the  AARs.  These  outcomes  are  precisely  what  define 
nonstructured  training. 

Thus,  by  making  changes  without  a  clear  understanding  of  the  linkages  inherent  in 
structured  training,  it  is  likely  that  a  nonstructured  exercise  will  be  generated,  using  some  of  the 
useful  products  of  the  structured  exercise.  The  value  of  the  products  will  be  lost  when  they  are 
disconnected  fi'om  other  components. 

Program  Implementation  Requirements 

This  lesson  is  related  to  the  balance  between  structure  and  flexibility  discussed  above.  It 
addresses  some  of  the  implementation  infrastructure  characteristics; 

Features  of  structured  training  programs  that  support  implementation  are 
there  for  a  reason  and  should  not  be  modified 

The  features  referred  to  include  distribution  of  materials,  qualifications  of  participants 
(especially  roleplayers),  training  for  simulation  interactors  and  workstation  roleplayers,  observer 
and  division  response  cell  preparations,  and  training  audience  preparation. 


59 


During  both  the  August  and  December  trial  implementations,  it  was  found  that 
participants  who  did  not  have  their  materials  prior  to  the  first  day  of  the  exercise  were  unlikely  to 
read  the  materials  (also  see  the  next  lesson  learned).  Those  who  did  have  the  materials  and  who 
read  them  found  them  useful.  The  conclusion  is  that  the  materials  must  be  distributed  in  time  for 
users  to  use  them. 

During  the  observed  implementations,  various  positions  at  roleplayed  workstations  were 
not  staffed  according  to  the  stated  requirements.  This  was  true  during  both  implementations  for 
one  or  more  battalion  workstations  and  for  the  division  response  cell  (EXCON)  workstation,  and 
was  observed  at  various  other  workstations  as  well.  When  this  happens,  participants  need  more 
instruction  at  the  beginning  and  continual  guidance  throughout  the  exercise.  Not  only  does  this 
disrupt  the  realism  of  the  input  to  the  brigade,  but  it  can  cause  developers  to  make  significant 
changes  to  TSPs  in  order  to  provide  sufficient  guidance  to  under-qualified  participants.  A  better 
solution  would  be  for  implementing  units  to  ensure  that  all  positions  are  filled  by  qualified 
personnel.  This  has  the  additional  advantage  of  delivering  valuable  incidental  training  to  persons 
who  can  best  benefit  from  it,  that  is,  incumbents  in  the  brigade’s  subordinate  and  supporting  units 
and  at  division  level. 

This  exercise,  like  other  unit  training  events,  requires  professionally  competent  soldiers  to 
deliver  the  appropriate  training  opportunities.  In  complex  group  performance  such  as  is  required 
of  a  brigade  staff,  no  TSP  alone  can  completely  replace  a  technically  competent  and  committed 
staff 

The  interactor  training  is  key  to  getting  the  designed  combat,  CS,  and  CSS  capabilities 
functioning  in  the  exercise.  Recognition  of  this  need  following  the  first  trial  implementation  led  to 
a  more  structured  training  session  for  the  second  trial.  Unfortunately,  the  training  was  not  used; 
the  simulation  site  staff  conducted  their  own  standard  BBS  training.  As  a  result,  interactors  were 
never  fully  familiar  with  the  goals  and  requirements  for  the  COBRAS  II  BSE.  One  solution, 
already  being  explored  in  other  projects,  is  to  present  the  interactor  training  in  a  less  directive 
mode,  listing  the  essential  skills  and  knowledges  for  BSE  interactors  rather  than  telling  the  site 
staff  how  to  ensure  that  the  information  gets  to  the  interactors.  A  related  need  concerns  training 
for  roleplayers  in  the  basics  of  BBS  capabilities.  When  roleplayers  do  not  spend  preparation  time 
with  their  workstation  interactor  team,  we  find  that  they  are  less  skillful  at  performing  the 
required  tasks,  including  troop  and  unit  movements,  reporting,  and  responding  to  brigade 
directives. 

Roleplayer  and  observer  participants  also  need  to  dedicate  time  to  becoming  familiar  with 
the  tactical  situation  that  underlies  the  exercise.  In  both  trial  implementations,  the  EXCON 
roleplayers  spent  considerable  time  (up  to  a  day)  pouring  over  the  division  OPORD,  presenting  a 
diwsion  OPORD  brief  to  the  observer  team  (roleplaying  the  brigade  staff),  and  walking  through 
the  messages  list  provided  in  the  EXCON  guide.  This  activity  was  invaluable  in  helping  both 
EXCON  and  the  observers  “get  their  heads  in  the  game.”  It  invariably  led  into  detailed 
discussions  of  the  tactical  situation,  the  exercise  flow,  OPFOR  activities,  and  brigade  staff 
performance  objectives. 

Finally,  training  audience  members  (i.e.,  the  brigade  staff)  need  to  be  reminded  that 
structured  programs  such  as  the  vignettes  and  the  BSE  offer  reduced  preparation  time,  not  zero 
preparation  time.  COBRAS  and  other  programs  with  detailed  TSPs  have  been  promoted  as  tum- 
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key  programs.  It  is  true  that  the  prepared  division  OPORDs  and  tactical  products  obviate  the 
need  to  develop,  wargame,  write,  and  produce  the  division  and  corps  OPORDs  and  tactical 
products,  which  are  the  major  cues  for  brigade  staff  planning.  The  prepared  messages  provide  the 
major  events  and  additional  cues  needed  to  make  the  scenario  successful.  However,  the  brigade 
staff  must  begin  its  participation  in  the  training  before  the  first  day  of  the  exercise,  with  the 
intelligence  preparation  of  the  battlefield,  logistics  preparation,  CP  setup,  and  review  of  their 
TACSOP.  Ideally,  they  would  also  meet  to  review  the  TACSOP  as  a  staff,  and  would  look  into 
the  doctrinal  references  listed  in  their  guides.  This  is  not  wasted  preparation  time-it  is  an  integral 
part  of  the  training. 


Training  Support  Products 

The  fourth  lesson  concerns  the  use  of  the  materials  and  training  products  contained  in  the 
TSP.  The  lesson  is  that: 

The  utility  of  the  TSP  will  be  limited  by  the  extent  to  which  it  is  used. 

This  is  (or  should  be)  no  surprise.  A  TSP  that  sits  on  a  shelf  or  is  not  read  will  not  be 
useful.  In  every  implementation,  the  simulation-based  scenario  tapes  or  files  and  the  tactical 
materials  are  always  used.  Indeed,  these  form  the  nucleus  of  the  exercise.  They  are  not  adopted 
wholesale;  Simulation  site  personnel  and  brigade  members  are  prone  to  alter  the  materials  and 
files  to  suit  what  they  perceive  to  be  a  unique  need  for  the  particular  implementation. 

Nonetheless,  they  invariably  realize  the  essential  nature  of  these  materials  and  files. 

The  remaining  materials-guides,  job  aids,  training  objectives,  AAR  materials,  and  so  on- 
tend  to  be  always  used  if  they  are  at  all  looked  at.  That  is,  the  simple  step  of  opening  the  guide 
and  reading  the  first  page  generally  ensures  that  the  user  will  read  most  of  the  remaining  pages, 
follow  the  guidance,  and  report  that  it  is  helpful. 

It’s  that  first  step-opening  the  materials  and  beginning  to  read  them-that  has  proved  to  be 
difficult  to  accomplish.  There  have  been  many  guesses  about  the  reasons-too  much  reading,  not 
enough  time,  materials  not  received  early  enough.  Given  that  those  who  do  start  to  read  the 
materials  usually  read  enough  to  find  them  useful,  that  first  reason  is  negated,  although  the  other 
two  may  be  valid. 

A  solution  to  the  difficulty  is  not  immediately  obvious.  It  has  been  suggested  that 
orientations  and  introductions  should  be  put  on  videotape  or  CD-ROM,  on  the  assumption  that 
users  will  learn  better,  or  at  least  be  drawn  into  the  materials,  if  their  first  exposure  is  aural  and 
nearly  passive.  However,  there  still  remains  the  first  step:  to  insert  the  tape  or  disk  into  the 
viewing  mechanism.  This  is  an  area  where  definition  of  the  problem  does  not  suggest  a  solution; 
more  study  and  experimentation  is  required. 

Nature  of  Lessons  Learned 

An  interesting  feature  of  this  report  and  other  lessons-leamed  reports  produced  by  ARI  is 
the  nature  of  the  lessons  learned,  which  itself  leads  to  a  lesson  learned: 

Documenting  global  lessons  learned  has  a  positive  effect  on  continued 
developmerU. 
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Every  one  of  the  lessons  learned  that  was  listed  in  the  COBRAS  I  report  was  found  to 
hold  true  for  the  COBRAS  11  experience.  In  many  cases,  we  have  been  able  to  incorporate 
lessons  in  a  next  generation  of  training  design  or  TSP.  For  example,  the  COBRAS  11  Exercise 
Guide  for  the  Exercise  Director  contains  guidance  on  how  to  modify  the  materials  for  different 
implementation  situations  (as  well  as  cautions  on  certain  types  of  changes). 

Other  lessons  are  doomed  to  be  learned  again  and  again.  The  lesson  cited  above 
concerning  utility  of  the  TSP  and  getting  participants  to  read  the  materials  has  always  been  a 
problem.  No  solution  yet  presents  itself  When  reading  the  training  program  instructions 
becomes  a  way  of  life  for  trainers,  we  will  have  taken  a  giant  step  toward  simplified  and 
standardized  implementation  of  trmning. 

Value  of  Structured  Training  Programs  and  Training  Support  Packages 

The  final  lesson  derived  from  the  COBRAS  n  development  and  implementation  concerns 
the  perceived  value  of  these  types  of  program: 

Having  tried  out  structured  training  and  TSPs,  units  see  the  value  of  the 
programs  as  part  of  a  training  strategy. 

During  the  course  of  the  BSE  trial  implementation,  participants  made  a  number  of 
remarks,  both  spoken  and  written,  that  provide  insights  into  the  value  of  structured  training 
programs  and  TSPs  like  those  provided  in  COBRAS  II.  The  statements  are  shown  in  Figure  35. 
These  statements,  together  with  our  observations  of  the  products  being  used  in  a  variety  of 
settings,  tell  us  that  there  is  a  requirement  for  structured  training  programs  that  focus  on  brigade 
combat  team  staff  performance.  There  are  two  fundamental  unit  needs  that  such  programs 
address:  the  need  for  continuous  performance  improvement,  and  the  need  for  readily  available 
training  packages. 

Performance  improvement  is  a  particular  concern  in  today’s  shrinking  force.  Brigade 
staffs  are  continuously  and  rapidly  turning  over  key  people.  Replacements  come  with  little  or  no 
experience,  and  usually  no  relevant  schooling  in  their  brigade  staff  role.  While  the  exercise  may 
initially  seem  to  address  very  fundamental  skills  of  planning,  preparation,  and  staff  processes,  in 
retrospect  this  seems  to  be  exactly  what  is  needed  at  the  early  stages  in  the  training,  maturation, 
and  development  of  these  “new  staffs.”  A  new  staff  that  runs  this  exercise  as  designed  will 
improve  its  ability  to  use  several  planning  and  preparation  staff  processes. 


62 


Type  of  *  ' 

Participant  . 

Training 

Audience 

BdeCdr:  We  need  more  TSPs  like  COBRAS.  COBRAS  allowed  me  to  maximize  my 

training  time  with  my  staff  with  very  little  overhead.  If  I  had  run  and  resourced  a 
BBS  without  COBRAS,  the  cost  would  be  prohibitive-  the  leaders  of  a  battalion- 
and  the  results  not  as  good. 

Bde  XO:  Great  training  value  at  very  little  cost  of  resources. 

Bde  S4:  This  is  exactly  what  I  needed  as  the  Bde  S4.  Been  in  the  job  3  months  and  this 
exercise  taught  me  a  lot.  Very  beneficial. 

MP  Pit  Ldr:  I  have  only  been  associated  with  the  Bde  for  about  3  weeks.  My  platoon  is 
being  “attached’Vassociated  w/  3d  Bde  solely  for  trainup  to  the  National  Training 
Center  mission.  This  exercise  was  extremely  helpful  for  me  to  see  where  I  fit  into 
the  Bde  picture.  This  was  the  first  chance  I’ve  had  to  work  with  the  Bde  staff. 

This  has  helped  to  define  my  role  in  the  planning  and  execution  process  of  the 

Bde.  Had  I  already  been  associated  with  the  Bde  and  worked  with  them  before, 
this  exercise  would  have  been  more  of  a  leadership  learning  experience. 

Exercise 

Director 

Really  happy  with  unit  use  of  your  products. .  .that’s  a  positive  sign  the  unit  thought  a  lot 
of  these  products.  I  am  impressed  with  flexibility  of  COBRAS’  design.  The 

AAR  booklet  is  real  good. 

Need  TSPs  for  Pit  through  Bde. 

COBRAS 

Coordinator 

A  very  positive  exercise  from  the  Brigade  perspective.  It  provided  a  new  staff  to  team- 
build  and  build  upon  new  processes  and  also  allowed  the  Brigade  Commander  to 
establish  standards.  I  believe  the  materials  are  sufficient  to  allow  a  staff  to  plan, 
prep  and  execute  this  exercise  without  much  difficulty. 

An  added  value  will  be  gained  once  more  variants,  in  terms  of  mission,  scenario,  and 
locations  are  developed.  This  Brigade  would  greatly  benefit  from  a  COBRAS 
exercise  in  a  Korean  scenario. 

Observers 

Would  love  to  have  TSPs  to  use  in  my  current  job:  Training  NG  enhanced  Bde. 

Good  exercise,  worth  refining  and  pushing  out  to  as  many  units  as  possible. 

Enjoyed  this  exercise  as  an  observer.  I  learned  a  lot.  I  would  definitely  want  to  do  this 
again. 

BBS  Site 
Manager 

More  time  devoted  to  training  for  COBRAS  than  a  normal  BBS  which  allowed  the  staff 
more  time  to  prepare,  and  overall  it  contributed  to  a  better  trained  staff. 

Figure  35.  Comments  from  COBRAS  II  Brigade  Staff  Exercise  participants  concerning  the  value 
of  structured  training  and  training  support  packages. 
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The  ready  availability  of  a  training  program  will  do  much  to  recommend  it  to  units.  Using 
prepared  TSPs  will  reduce  the  staff  time  and  energy  needed  to  prepare  for  and  execute  a 
structured  training  exercise.  During  the  exercise,  the  TSP  provides  the  conditions  and  specific 
cues  that  prompt  staff  actions.  These  conditions  and  cues  are  carefully  cross-checked  and 
integrated  to  provide  doctrinally  correct  activities  which  permit  the  staff  to  focus  on  its  processes 
rather  than  on  the  tactics  of  the  exercise. 

The  statements  also  validate  implementation  decisions  concerning  the  length  of  the 
training  day.  The  exercise  schedule  supports  frequent  feedback  and  provides  the  staff  the  time  to 
complete  all  its  required  activities,  participate  in  feedback  sessions,  interact  with  subordinates,  and 
be  mentally  fresh  enough  to  master  complex  cognitive  processes. 

Summary 

This  section  has  presented  and  discussed  some  of  the  major  lessons  learned  from  the 
COBRAS  n  development  and  formative  evaluation  effort.  Some  of  the  lessons  were  based  on 
direct  feedback  from  pilot  test  and  trial  implementation  participants,  while  others  were  based  on 
observations  of  the  implementations.  Lessons  were  noted  during  all  phases  of  the  project,  from 
the  initial  design  through  implementations. 

The  final  section  in  this  report  describes  some  issues  that  remain  to  be  addressed.  Some, 
such  as  the  COBRAS  HI  program,  are  underway  already.  Others  will  be  likely  to  demand 
attention  in  the  near  (5  year)  future,  if  the  U.S.  Army  is  to  continue  with  steady  progress  toward 
its  Force  XXI  goals  in  training  and  readiness. 
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SECTION?:  FUTURE  DIRECTIONS 


Even  as  the  COBRAS  development  and  refinement  was  being  completed,  additional 
structured  training  R&D  programs  had  begun.  They  are  designed  to  follow  up  on  additional 
training  needs  for  brigade  and  below  that  were  pointed  out  by  users,  observed  by  the  COBRAS 
developers,  or  suggested  by  the  Force  XXI  Training  program  and  other  stakeholders. 

The  most  pressing  needs  that  had  been  identified  for  development  are  listed  below:  The 
list  is  formidable,  and  only  a  few  of  these  identified  needs  are  likely  to  be  addressed  anytime  soon. 
Those  ARI-AFRU  projects  that  are  already  underway  or  designated  for  imminent  commencement 
are  identified  by  a  project  code  name  following  the  description  (code  names  are  identified  with 
ARI-AFRU  project  titles  in  Appendix  A). 

•  A  second  BSE,  on  a  different  terrain  data  base; 

•  An  expanded  multiechelon  version  of  the  BSE,  still  in  BBS  {COBRAS  ///); 

•  A  multiechelon  BSE  incorporating  linked  virtual  (SIMNET  and  reconfigurable 
simulations)  and  constructive  (BBS)  simulation  environments  (COBRAS  III), 

•  Expanded  training  for  members  of  brigade  and  battalion  staff  sections  (COBRAS  III); 

•  More  flexible  implementation  models  that  allow  24-hour  operations,  deployed  CPs, 
more  intense  battle  rhythms,  and  more  robust  and  aggressive  OPFOR  activity 
(COBRAS  III); 

•  A  parallel  version  of  the  BSE  for  light  and  light-heavy  units; 

•  A  version  of  the  BSE,  incorporating  CSS  emphasis  and  the  full  planning  and 
preparation  phases,  for  maneuver  battalions; 

•  Additional  vignettes  focused  on  maneuver  battalion  training  needs; 

•  A  supported  implementation  of  a  sequence  of  Force  XXI  programs  with  more 
complete  assessment  of  implementation  requirements  and  training  outcomes  (ISAT); 

•  Development  of  structured  simulation-based  training  exercises  using  the  Close  Combat 
Tactical  Trainer  (STRUCCTT,  STRUCCTTII);  and 

•  Development  of  an  automated  tool  to  help  commanders  and  unit  trainers  access  TSPs, 
modify  existing  scenarios  and  TSPs,  or  create  their  own  scenarios  and  TSPs  (C/77). 

The  Next  Step:  COBRAS  HI 

As  shown  above,  the  COBRAS  III  work  represents  an  expansion  to  the  COBRAS  11  BSE 
in  several  areas,  including  the  training  audience,  implementation  model,  and  simulation 
environment.  COBRAS  HI  comprises  two  major  components:  a  constructive  simulation  Brigade 
and  Battalion  StaffExercise  (BBSE)  and  a  linked  constructive  and  virtual  simulation  exercise 
known  as  the  Synthetic  Theater  of  War  Exercise. 

The  BBSE  is  incorporating  several  innovations  in  addition  to  the  expanded  training  for 
brigade  and  battalion  staff  sections,  24-hour  operations,  deployed  CPs,  more  intense  battle 
rhythms,  and  more  robust  and  aggressive  OPFOR.  One  of  these  concerns  an  often  observed  and 
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voiced  expectation  that  the  BSE  was  most  valuable  as  an  NTC  ramp-up  exercise.  This  led  to 
some  sweeping  and  often  confusing  modifications  in  implementation,  as  witnessed  in  the  May 
1997  implementation  at  Fort  Lewis  (3  Bde,  2  ID).  Among  the  modifications  were  changes  in  task 
organization  (use  of  one  armor,  one  mechanized  infantry,  and  one  light  infantry  battalion), 
schedule  (overlapping  missions  and  concurrent  planning  and  operations,  rather  than  sequential 
missions),  and  OPFOR  activity  and  aggressiveness.  All  of  these  changes  were  intended  to  provide 
a  virtual  replication  of  the  anticipated  NTC  experience. 

The  BSE  was  intended  as  a  “crawl”  level  exercise,  to  help  brigade  staff  members  learn 
about  their  own  jobs  wthin  the  larger  staff  process,  to  allow  them  to  practice  interactions  and 
information  flow,  and  to  give  them  experience  in  using  all  of  their  assets-combat,  CS,  and  CSS. 
The  May  implementation  was  an  attempt  to  adapt  this  exercise  as  a  “graduate-level”  exercise, 
preparatory  to  their  NTC  rotation.  More  than  words,  these  actions  expressed  the  brigade’s  need 
for  simulation-based  practice  opportunities  that  would  help  to  prepare  them  for  a  high-intensity, 
realistic  field  exercise  and,  by  extension,  for  a  real  world  mission-required  deployment. 

The  COBRAS  III  BBSE  is  providing  a  portion  of  the  solution  to  that  training  need.  The 
BBSE  will  not  be  a  replica  of  any  one  NTC  or  other  CTC  rotation.  Rather,  it  will  incorporate 
some  of  the  features  that  provide  valuable  training,  including  24-hour  operations,  deployed  CPs, 
concurrent  operations  (overlapping  missions).  The  training  audience  is  the  brigade  commander, 
his  primary  and  special  staff  and  their  sections,  the  maneuver  commanders,  and  their  primary 
staffs  and  sections.  Training  audience  members  are  not  specifically  named  (as  in  COBRAS  11), 
but  the  brigade  and  battalion  leaders  are  to  be  inclusive  in  who  they  bring  to  the  exercise  and  to 
the  AARs. 

Because  the  BBSE  is  not  expected  to  be  implemented  with  a  dedicated  0/C  team,  does 
not  use  extensive  real  terrain,  and  does  not  have  sophisticated  data  collection  instrumentation  (as 
at  the  CTCs),  it  can  never  be  a  “CTC-in-a-box.”  Rather,  it  is  designed  to  provide  home  station 
training,  prior  to  CTC  rotation,  that  will  enable  a  brigade  to  get  more  training  value  out  of  its 
high-cost  CTC  tr^ning. 

Another  feature  of  the  COBRAS  III  BBSE  that  is  aimed  at  enhancing  the  multiechelon 
aspect  of  the  training  and  preparing  the  brigade  and  its  battalions  for  the  CTC  (or  deployment) 
lies  in  its  designation  of  the  program  training  objectives.  Rather  than  focusing  on  procedural 
activities  and  brigade  products  and  outcomes  (as  did  the  COBRAS  II  BSE),  the  training 
objectives  are  centered  on  techniques  and  procedures  that  give  the  brigade  an  edge  in  performing 
key  high-payoff  tasks.  For  example,  one  focus  is  on  how  the  brigade  and  battalion  CPs  manage 
information  throughout  all  phases  of  a  mission.  Another  focus  is  on  the  planning  and  management 
of  the  reconnaissance  and  surveillance  for  the  brigade.  Yet  another  concerns  how  the  brigade  and 
battalions  manage  their  parallel  planning  efforts.  The  materials  for  the  training  audience  and  the 
observers  will  contain  descriptions  of  ways  to  enhance  their  performance  of  these  key  tasks, 
derived  fi^om  the  experiences  of  combat  veterans  and  CTC  observers  (especially  as  documented 
by  the  Center  for  Army  Lessons  Learned  [CALL]  at  Fort  Leavenworth). 

A  second  effort  within  COBRAS  III  is  exploring  the  use  of  linked  constructive  and  virtual 
simulations  for  multiechelon  training.  The  prototype  TSP  is  being  designed  for  a  one-time 
implementation  at  Fort  Knox  in  early  spring  1998.  Products  will  include  both  the  partially 
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complete  TSP  and  complete  documentation  of  the  remaining  developmental  needs  for  both  the 
TSP  and  the  simulation  itself 


Summary 

The  purpose  of  this  report  was  to  describe  the  development  of  the  COBRAS  n  brigade- 
level  training  exercises  and  to  offer  lessons  learned  to  the  military  training  development 
community.  The  report  began  with  a  project  introduction  focusing  on  the  project’s  background  in 
terms  of  its  relationship  to  COBRAS  I.  It  then  identified  the  project’s  objectives,  scope,  and 
development  methodology. 

The  formative  evaluation  strategy  was  then  presented.  It  described  the  concept  of 
formative  evaluation  and  the  methods  for  quality  assurance  that  were  utilized  during  the  project. 
This  section  also  presents  the  COBRAS  n  developmental  timeline. 

The  report  described  the  COBRAS  11  BSE  design  parameters  and  the  thought  processes 
used  to  design  the  program.  In  the  same  section,  the  report  discussed  the  identification  of  training 
objectives  through  the  ModSPA  process  and  the  development  of  the  scenario.  Because  the 
COBRAS  n  BSE  was  constructed  as  an  expansion  and  enhancement  of  the  COBRAS  I  exercise, 
most  of  the  design  and  development  description  is  in  terms  of  how  the  original  work  was  adapted. 
The  COBRAS  I  work  and  products  are  themselves  covered  in  Appendix  B. 

One  section  then  detailed  the  major  formative  evaluation  activity  for  the  BSE,  the  trial 
implementation  in  Fort  Lewis  with  3  Bde,  2  ID  in  December  1996.  This  section  also  contained 
detailed  summaries  of  the  formative  evaluation  data  collection  and  descriptions  of  the  TSP  and 
scenario  modifications  that  resulted. 

A  single  section  contains  details  about  COBRAS  II  vignette  design,  development,  pilot 
implementation,  and  quality  assurance  reviews.  As  with  the  BSE,  vignette  development  processes 
were  modeled  on  the  COBRAS  I  efforts.  One  significant  outcome,  in  addition  to  the  vignette 
TSPs,  is  a  methodology  guide  for  development  of  small  group  staff  exercises  such  as  the 
vignettes. 

The  report  concludes  by  presenting  lessons  learned  during  the  course  of  the  project  and 
discussing  fiature  development  needs.  The  lessons  have  a  general  applicability  and  highlight  the 
important  issues  that  surfaced  during  the  program’s  design,  development,  and  formative 
evaluation.  Several  of  the  suggested  expansions  to  the  program  are  already  underway  in 
COBRAS  m  and  ISAT. 
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APPENDIX  A 


ACRONYMS  and  ABBREVIATIONS 


A2C2 

army  airspace  command  and  control 

AA 

assembly  area 

AAR 

after  action  review 

AC 

active  component 

AD 

area  defense 

ADA 

air  defense  artillery 

ADCOORD 

air  defense  coordinator 

AGMB 

Advance  Guard  Main  Body 

ARI 

Army  Research  Institute 

ARTEP 

Army  Training  and  Evaluation  Program 

AVNLNO 

aviation  liaison  officer 

BBS 

Brigade/Battalion  Battle  Simulation 

BBSE 

Brigade  and  Battalion  Staff  Exercise 

Bde 

brigade 

BG 

brigadier  general 

BOS 

battlefield  operating  system 

BSE 

Brigade  Staff  Exercise 

C3 

command,  control,  and  communication 

CALL 

Center  for  Army  Lessons  Learned 

Cdr 

commander 

CHEMO 

chemical  officer 

CITT 

Commander’s  Integrated  Training  Tool  for  the  Close  Combat  Tactical  Trainer 

Co 

company 

COA 

course  of  action 

COBRAS 

Combined  Arms  Operations  at  Brigade  Level,  Realistically  Achieved  Through 
Simulation 

COL 

colonel 

CP 

command  post 

CS 

combat  support 

CSS 

combat  service  support 

CTC 

Combat  Training  Center 

DATK 

DDMP 

DIV 

DS 
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Department  of  the  Army 
deliberate  attack 

deliberate  decision-making  process 

division 

direct  support 


ENG 

engineer 

EXCON 

Exercise  Control 

FASCAM 

Family  of  Scatterable  Mines 

FM 

field  manual 

FSB 

forward  support  battalion 

FSO 

fire  support  oflBcer 

FSCOORD 

fire  support  coordinator 

FIX 

field  training  exercise 

FXXITP 

Force  XXI  Training  Program 

G1 

division  adjutant 

G2 

division  intelligence  officer 

G3 

division  operations  officer 

G4 

division  supply  officer 

HICON 

Higher  Control 

HumRRO 

Human  Resources  Research  Organization 

ID 

infantry  division 

IPB 

Intelligence  Preparation  of  the  Battlefield 

IPR 

in-progress  review 

ISAT 

Implementation  Support  and  Assessment  Team 

Ldr 

leader 

Lie 

lieutenant  colonel 

METT-TC 

mission,  enemy,  terrain,  troops,  time  available,  and  civilian  considerations 

MG 

major  general 

MI 

military  intelligence 

ModSPA 

modified  staff  performance  analysis 

MP 

military  police 

MTC 

movement  to  contact 

MTP 

Mission  Training  Plan 

NCO 

noncommissioned  officer 

NG 

national  guard 

NIC 

National  Training  Center 

QIC 

officer  in  charge 

0/C 

observer/controller 

OPFOR 

opposing  force 

OPORD 

operations  order 

OPS 

operations 
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PAM 

pamphlet 

Pit 

platoon 

R«&D 

research  and  development 

RTO 

radiotelephone  operator 

SI 

adjutant 

S2 

intelligence  officer 

S3 

operations  and  training  officer 

S4 

supply  officer 

SIGO 

signal  officer 

SIMNET 

Simulation  Networking 

SME 

subject  matter  expert 

SOP 

standing  operating  procedure 

SOW 

statement  of  work 

SPA 

staff  performance  analysis 

STARTEX 

start  of  exercise 

STRUCCTT 

Structured  Training  for  Units  in  the  Close  Combat  Tactical  Trainer 

TAG 

tactical  command  post 

TACSOP 

tactical  standing  operating  procedure 

TF 

task  force 

TOC 

tactical  operations  center 

TOE 

table  of  organization  and  equipment 

TRADOC 

U.S.  Army  Training  and  Doctrine  Command 

TSP 

training  support  package 

USAARMC 

U.S.  Army  Armor  Center 

USAARMS 

U.S.  Army  Armor  School 

VTP 

virtual  training  program 

WARNO 

warning  order 

XO 

Executive  Officer 
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FOUNDATIONS:  OVERVIEW  OF  THE  COBRAS  I  PROGRAM 

The  original  Combined  Arms  Operations  at  Brigade  Level,  Realistically  Achieved  Through 
Simulation  (COBRAS)  program  consisted  of  a  Brigade  Staff  Exercise  (BSE)  and  a  set  of  brigade 
staff  vignettes.  It  was  developed  at  Fort  Knox,  Kentucky  by  the  U.  S.  Army  Research  Institute 
for  the  Behavioral  and  Social  Sciences’  (ARI)  Armored  Forces  Research  Unit,  with  the  assistance 
of  the  same  consortium  of  contractors  that  was  responsible  for  developing  the  expansion.  The 
COBRAS  I  exercises  provided  the  models  from  which  the  expansion  exercises  would  be  designed, 
and  in  this  way,  served  as  a  test-bed  for  furthering  the  assembly  of  the  Force  XXI  training 
regimen. 

This  appendix  provides  a  concise  description  of  the  COBRAS  I  exercises  and  some 
discussion  of  key  development  processes.  The  intent  is  to  provide  the  background  necessary  for 
understanding  the  COBi^S  n  project’s  development  processes  and  outcomes,  given  that  most  of 
these  processes  and  outcomes  were  based  on  activities  of  the  COBRAS  I  project.  The  two  major 
topics  of  the  section  are: 

•  The  COBRAS  I  Brigade  Staff  Exercise:  Provides  a  description  of  the  original  BSE 
and  key  development  processes,  including  the  development  of  training  objectives. 

•  The  COBRAS  I  Brigade  Staff  Vignettes:  Provides  a  description  of  the  original  set  of 
vignettes  as  they  existed  at  the  conclusion  of  the  COBRAS  I  project  and  presents  the 
key  development  processes. 

The  COBRAS  I  Brigade  Staff  Exercise 

The  BSE  is  a  multi-mission,  large-scope  practice  exercise  that  focuses  on  the  interactions 
among  the  brigade  commander  and  his  staff  as  they  conduct  planning  and  employ  brigade  assets. 
This  focus  was  selected  due  to  indications  that  the  brigade  commander  and  his  staff  need 
structured  practice  opportunities  to  achieve  proficiency  in  basic  brigade  operations  of  planning 
and  synchronizing  assets.  The  program,  as  designed,  gives  the  commander  and  his  staff  a  chance 
to  practice  the  tasks  they  should  perform  as  they  fight  the  brigade  in  the  particular  battles  of  a 
structured  scenario.  Within  a  simulated  Brigade/Battalion  Battle  Simulation  (BBS)  combat 
situation,  they  must  determine  what  has  to  be  done  on  the  battlefield,  who  does  it,  and  how  their 
actions  are  linked  to  actions  of  other  units  and  battlefield  operating  systems  (BOS). 

This  description  of  the  BSE  describes  four  key  aspects  of  the  exercise: 

•  the  training  audience, 

•  the  training  objectives  and  their  development, 

•  the  exercise  implementation  model,  and 

•  the  structure  and  composition  of  the  training  support  package  (TSP)  that  includes  all 
the  information  required  to  implement  the  exercise. 
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Training  Audience 

The  target  training  audience  was  operationally  defined  as  those  participants  for  whom 
training  objective  task  lists  would  be  generated,  observers  would  be  assigned,  and  after  action 
review  (AAR)  sessions  would  be  provided.  The  target  audience  included  the  brigade  commander, 
the  brigade  primary  staff  (i.e.,  Executive  OflSce  [XO],  Adjutant  [SI],  Intelligence  Office  [S2], 
Operations  and  Training  Officer  [S3],  Supply  Officer  [S4],  and  Fire  Support  Officer  [FSO]),  and 
the  special  staff  who  serve  as  links  between  the  brigade  and  four  of  its  systems  (i.e.,  fire  support, 
air  defense,  engineering,  and  logistics).  These  linking  personnel  were  determined  to  be  the 
following: 

•  the  air  defense  coordinator  (ADCOORD),  the  air  defense  artillery  (ADA)  battery 
commander, 

•  the  engineer  (ENG)  battalion  commander, 

•  the  forward  support  battalion  (FSB)  commander,  and 

•  the  fire  support  coordinator  (FSCOORD),  the  direct  support  (DS)  artillery  battalion 
commander. 

Training  Objectives  and  Tasks 

One  of  the  most  defining  features  of  the  COBRAS  I  BSE  is  its  set  of  exercise  performance 
objectives  and  tasks.  With  a  focus  on  the  planning  and  synchronization  of  brigade  assets,  as  well 
as  a  special  emphasis  on  combat  service  support  (CSS)  fimctions,  the  BSE  performance  objectives 
cover  a  wide  range  of  staff  activities.  These  activities  are  summarized  in  four  general  staff 
performance  objectives: 

1.  Performance  of  the  full  mission  requirements  of  planning,  preparation,  and  execution 
(including  consolidation,  reorganization,  and  planning  for  follow-on  missions). 

2.  Performance  of  both  the  deliberate  decision-making  process  (DDMP^,  performed 
without  time  pressure,  and  a  modified  decision-making  process,*®  performed  under 
time  constrained  conditions. 

3.  Complete  production  of  planning  and  preparation  products,  including  interim  products 
and  inputs. 

4.  Integration  of  selected  combat  support  (CS)  and  CSS  functions  into  the  staff  processes 
of  planning,  preparation,  and  execution. 

These  objectives  are  supported  by  arrays  of  brigade  staff  tasks  that  are  specified  for  each 
of  the  1 1  members  of  the  target  training  audience  for  each  of  the  three  missions-a  total  of  33  task 


®  During  the  period  of  time  in  which  COBRAS  I  development  proceeded,  the  decision-making  process  was  titled 
“Deliberate  Decision-Making  Process”  (DDMP).  Later,  during  the  COBRAS  II  project,  the  name  was  officially 
changed  to  the  “Military  Decision-Making  Process.” 

The  process  called  “Modified  Decision-Making”  in  the  COBRAS  I  work  was  referred  to  variously  as  the 
abbreviated  or  accelerated  decision-making  process  in  Field  Manual  (FM)  71-3  (Department  of  the  Army  [DA], 
1995)  and  the  combat  decision-making  process  in  FM  101-5  (DA,  1997). 
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lists.  The  tasks  are  consistent  with  current  doctrine,  as  defined  by  Army  manuals  such  as  Army 
Training  and  Evaluation  Program  (ARTEP)-Mission  Training  Plan  (MTP)  and  Field  Manual 
(FM)  publications,  but  are  not  constrained  to  the  contents  of  these  documents.  Rather,  the  tasks 
include  living,  innovative,  and  adaptive  tasks,  which  are  descriptions  of  behaviors  that  underlie 
successful  and  exemplary  performance.  During  the  project,  the  cumulative  domain  of  these 
behaviors  was  termed,  “undocumented  tasks,”  to  differentiate  them  from  the  mainstream, 
primarily  ARTEP-based,  documented  tasks. 

The  methodology  used  to  identify  the  COBRAS  tasks  was  entitled  the  Staff  Performance 
Analysis,  or  SPA.  The  methodology  employed  roleplay  enactments  of  the  BSE’s  missions  and 
subsequent  probing  activities  to  determine  the  activities  of  the  commander  and  staff  during  the 
BSE’s  particular  missions.  Tasks  identified  during  the  SPA  met  certain  criteria  based  on  project 
objectives.  These  criteria  included  the  following: 

•  Content  relevance:  Performance  was  defined  by  the  mission,  enemy,  terrain,  troops, 
time,  and  civilian  considerations  (METT-TC)  of  three  specific  missions.  Unlike 
existing  generalizable  performance  descriptions,  the  SPA  and  its  outcomes  (i.e.,  tasks) 
were  to  be  specific  to  COBRAS  METT-TC. 

•  Functional  emphasis:  Developers  documented  the  processes  during  each  mission 
phase  (planning,  preparation,  executing  the  mission,  and  conducting  consolidation  and 
reorganization)  in  specific  functional  areas:  command  and  control,  CSS,  fire  support, 
air  defense,  mobility/countermobility/survivability,  intelligence,  and  maneuver.  The 
cells  in  the  matrix  formed  by  the  overlay  of  these  two  domains  (mission  phase  X 
functional  area)  were  to  serve  as  a  structure  for  identifying  tasks. 

•  Interactive  criticality:  Special  emphasis  was  placed  on  identifying  activities  which 
require  interactions  between  pairs  or  groups  of  training  audience  members. 

•  Observability:  Tasks  were  to  have  observable  processes,  products,  or  outcomes. 
Because  the  training  process  emphasized  the  contributions  of  observers  who  would 
coach  and  provide  feedback,  it  was  necessary  that  observable  task  indicators  be 
identified.  Important  as  they  are,  performance  requirements  such  as  reasoning, 
thought  processes,  and  knowledge  cannot  be  readily  coached  and  evaluated  during 
performance  unless  there  are  observable  outcomes. 

•  Brigade  staff  process  focus:  The  focus  of  performance  for  the  training  participants  is 
only  on  their  roles  within  the  conduct  of  staff  processes.  Tins  is  especially  significant 
in  the  situations  of  those  individuals  who  have  command  responsibilities  in  addition  to 
brigade  staff  responsibilities.  For  example,  the  brigade  ENG  is  also  the  commander  of 
the  engineer  battalion,  but  tasks  involving  the  command  of  his  battalion  were  not  to  be 
addressed. 

The  Exercise  Implementation  Model 

The  implementation  model  can  be  defined  in  terms  of  four  exercise  aspects:  participants, 
simulation  site  layout,  preparation  for  the  exercise,  and  conduct  of  the  exercise.  The  discussion 
below  describes  these  aspects  to  depict  how  the  BSE  is  implemented. 
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The  exercise  participants.  The  BSE,  as  designed,  requires  a  total  of  102  participants,  in 
addition  to  the  1 1  included  in  the  target  training  audience.  These  participants  fill  roles  ranging 
from  high-level  exercise  management,  to  the  roleplaying  of  subordinate  unit  staffs,  to  operating 
BBS.  Additional  participants  may  be  inserted  into  any  of  the  role  types,  but  decreasing  the 
exercise  staff  may  reduce  the  training  benefit  of  the  brigade  staff.  The  types  and  positions  of  the 
participants  prescribed  for  implementation  are  provided  in  Figure  B-1.  Figure  B-2  presents  brief 
descriptions  of  each  participant  type. 


Figure  B-1.  Locations  of  exercise  participants  in  the  COBRAS  I  Brigade  Staff  Exercise. 
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Role 

Exercise 

Management 

Senior  military  personnel  fill  the  exercise  management  positions. 

•  The  Exercise  Director  provides  overall  leadership  and  is  involved  in  every  decision 
that  may  affect  the  ability  of  the  training  exercise  to  meet  the  training  objectives. 

•  The  Exercise  Director’s  primary  assistant,  the  COBRAS  Coordinator,  coordinates 
and  arranges  for  personnel,  supplies,  and  equipment  in  preparation  for  the  training. 

The  COBRAS  Coordinator  is  also  available  during  training  to  assist  the  Exercise 
Director  and  to  keep  him  informed  of  the  course  the  training  is  taking. 

•  The  Blue  Forces  Controller  monitors  all  activities  regarding  the  operation  of  the 
simulation  for  Blue  Force  units. 

Observers 

The  observers  provide  feedback  to  the  target  training  audience  regarding  their 
performance.  Feedback  is  provided  in  a  series  of  after  action  review  (AAR)  sessions  and 
during  execution  in  the  form  of  the  coaching  of  individual  training  audience  members. 

The  Senior  Observer  is  the  leader  of  the  observer  team.  His  responsibilities  include 
facilitating  AARs,  adjusting  observer  assignments,  and  serving  as  an  advisor  to  the 

Exercise  Director  on  how  the  exercise  is  going  and  implementing  solutions. 

Roleplayers 

Subordinate  and  Supporting  Units:  The  major  elements  of  the  brigade  are  portrayed  by 
personnel  of  the  brigade’s  subordinate  and  supporting  units,  performing  the  duties  of  the 
functions  and  imits  to  which  they  are  normally  assigned. 

•  Exercise  Control  (EXCON):  EXCON  roleplayers  play  the  roles  of  key  division  staff 
positions.  They  dispense  scripted  and  hard  copy  messages  intended  to  maintain  the 
integrity  and  flow  of  the  scenario  to  support  the  training  objectives.  In  doing  so,  the 
EXCON  roleplayers  cause  the  occurrence  of  significant  events  that  cue  staff  actions. 

In  addition,  the  EXCON  roleplayers  respond  to  questions  and  requests  from  the 
brigade  staff  using  prepared  guidelines. 

•  Opposing  Force  (OPFOR):  The  OPFOR  Controller  is  the  roleplayer  who  directs  the 
actions  of  the  OPFOR  following  the  intent  of  the  exercise  and  situation-specific 
guidelines.  His  purpose  is  to  provide  the  cues  to  stimulate  performance  of  the 
training  objectives  by  the  brigade  staff.  He  works  with  Brigade/Battalion  Battle 
Simulation  (BBS)  interactors  at  three  workstations. 

BBS 

Interactors 

Interactors  operate  the  computer  terminals  that  control  BBS.  Their  role  is  to  translate  the 
tactically-oriented  instructions  provided  by  roleplayers  into  BBS  computer  commands. 
Interactors  also  construct  and  print  out  tactical,  logistical,  and  status  reports  of  the  units. 

Brigade  Staff 
Support 

These  personnel  fill  the  positions  they  normally  occupy  to  support  brigade  staff  functions. 
They  include  staff  assistants,  radiotelephone  operators,  etc. 

HICON 

The  Higher  Control  (HICON)  Operator,  who  operates  the  HICON  workstation,  performs 
certain  simulation  operations  that  are  not  executable  by  roleplayers  and  interactors 
located  at  other  workstations.  The  HICON  Operator  is  a  member  of  the  simulation  site 
staff.  He/she  designates  other  members  of  the  staff  to  conduct  interactor  and  roleplayer 
training  on  BBS  and  to  oversee  and  facilitate  overall  operation  of  the  simulation. 

Figure  B-2.  Role  descriptions  of  COBRAS  I  Brigade  Staff  Exercise  participant  types. 
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Simulation  site  layout.  As  stated  above,  the  BSE  is  implemented  within  the  confines  of 
the  BBS.**  The  training  is  conducted  using  three  simulated  command  post  (CP)  locations  (the 
tactical  [TAC]  CP,  the  main  CP,  and  the  rear  CP)  for  the  brigade  staff  and  either  10  or  14  BBS 
workstations.  All  CPs  and  BBS  workstations  are  linked  by  radio  communications  representing 
eight  brigade  nets. 

During  BSE  execution,  the  participants,  including  the  primary  training  audience,  are 
located  according  to  their  functions  in  the  exercise,  and  as  required  by  the  simulation  layout. 

Their  placement  was  dependent  upon  satisfying  the  notion  that  they  should  be  in  the  location  that 
best  facilitates  the  performance  of  their  roles.  Figure  B-3  portrays  a  typical  simulation  center 
setup  and  layout  of  work  areas  for  the  BSE,  and  indicates  the  locations  of  the  participant  types. 
Note  that  the  exercise  management  personnel  do  not  have  designated  locations  from  which  to 
work.  The  Exercise  Director  usually  spends  the  majority  of  his/her  time  in  the  CPs  or  Exercise 
Control  (EXCON)  workstation,  and  the  Blue  Forces  Controller  spends  most  of  his/her  time  in  and 
around  the  subordinate  and  supporting  unit  workstations. 

Preparation  for  the  exercise.  The  planning  and  preparation  for  an  implementation  of  the 
BSE  begins  wth  the  brigade  learning  about  and  deciding  to  conduct  the  exercise,  entering  the 
exercise  in  their  training  calendar,  and  scheduling  the  BBS  facility  for  the  exercise.  All 
subsequent  activities  in  this  phase  should  begin  approximately  18  weeks  prior  to  the  exercise 
conduct.  The  planning  and  preparation  process  timeline  shown  in  Figure  B-4  is  extracted  from 
the  Exercise  Guide,  a  key  portion  of  the  TSP-described  below.  It  indicates  the  primary  decision 
and  coordination  activities  that  should  occur  during  the  weeks  prior  to  actual  implementation. 

Conduct  of  the  exercise.  Conduct  of  the  BSE  requires  the  brigade  staff,  exercise 
management  personnel,  and  roleplayers  to  be  familiar  with  the  tactical  situation  of  the  first  mission 
to  be  conducted,  and  the  simulation  and  its  operators  (i.e.,  interactors)  to  be  prepared  for  their 
functions.  The  initial  event  in  the  exercise  is  the  delivery  of  a  division  order  by  the  EXCON 
roleplayers  and  Exercise  Director.  This  event  sets  the  brigade’s  planning  process  in  motion. 

The  activities  of  the  exercise  conduct  include  the  planning,  preparation  and  execution 
phases  of  the  mission(s),  and  a  series  of  AARs  that  are  organized  to  address,  in  a  sequential 
manner,  the  segments  of  the  mission(s)  conducted.  Thus,  the  prime  determinant  of  the  content  of 
the  exercise  is  the  mission(s)  selected  for  the  exercise.  Depending  on  the  mission(s)  selected,  the 
brigade  will  train  either  on  the  DDMP,  the  modified  decision-making  process,  or  both.  The 
scenario  allows  for  six  viable*^  mission  implementation  options,  catering  to  the  training  needs  of 
individual  brigades.  The  six  viable  options  are  portrayed  in  Figure  B-5. 


”  This  simulation  was  chosen  over  Janus,  Simulation  Networking  (SIMNET),  and  distributed  interactive 
simulations  (i.e.,  an  environment  in  which  either  Janus  or  BBS  is  linked  electronically  to  SIMNET)  because  of 
its  capabilities  to  satisfy  five  criteria:  functional  representation,  the  size  of  the  terrain  database,  the  ability  to 
generate  combat  report  information,  operator  requirements,  and  brigade  asset  representation. 

Other  performance  sequences  are  illogical  because  they  do  not  support  the  concepts  in  which  the  program  is 
grounded,  including  leveraged  use  of  simulation  and  a  focus  on  designated  training  objectives. 
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Brigade  Staff  Work  Areas 
A  -  Main  CP 
B  -  Rear  CP 

C  -  TAC  CP  (Execution  only) 


HICON/EXCON  BBS  Room 

4  -  HICON 

5-EXCON 


Red  Forces  BBS  Room 

1  -  OPFOR  BBS  Station  1 

2  -  OPFOR  BBS  Station  2 
3 -OPFOR  BBS  Station  3 


A  BBS  ROOM 

a  -  Combat  terminal 
b  -  CS  terminal 
c  -  CSS  terminal  ^ 


Blue  Forces  BBS  Room 

6  •  Field  artillery  battalions 

7  -  Engineer  battalion 
8-11  -  Taskforces 


<D  ® 


12  -  Cavalry  troop 

13  -  Air  defense  artillery  battalion 

14  -  Brigade  support  area  and  forward  support  battalion 


Figure  B-3.  Layout  of  work  areas  for  the  Brigade  Staff  Exercise. 
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T-18 

weeks 


Divisicm  Operations  (G3)  designates  personnel  to  serve  as  Exercise  Director  and  COBRAS 
Coordinator. 

Exercise  Director  and  Brigade  Commander  meet  to  discuss  roles  and  expectations. 


T-14 

weeks 


Exercise  Director  and  Brigade  Commander  select  option  for  mission(s)  to  conduct. 
Exercise  Director  develops  preliminary  exercise  schedule. 


T-12 

weeks 


COBRAS  Coordinator  prepares  taskings  for  personnel;  released  to  units. 
COBRAS  Coordinator  confirms  fecilities  schedule. 


T-4  weeks 


COBRAS  Coordinator  issues  participant  guides  and  readahead  materials  to: 

•  Training  audience  (through  Brigade  XO) 

•  Exercise  Control  (EXCON)  G3  Roleplayer 

•  Opposing  Force  (OPFOR)  Controller 

•  Observers 


T-2  weeks 


COBRAS  Coordinator  issues  guides  and  read-a-head  materials  to  Blue  Forces  roleplayers. 
Simulation  site  personnel  load  and  try  out  Brigade/Battalion  Battle  Simulation  (BBS)  tapes. 
Orientation  briefing  for  all  participants. 

Brigade  Commander  and  staff  study  readahead  materials  and  references  and  get  familiar 
with  the  tactical  situation. 

Brigade  Commander  and  XO  decide  on  support  staffing. 

Brigade  XO  arranges  for  copies  of  the  overlays  to  be  made. 

Brigade  Intelligence  Officer  (S2)  uses  readahead  materials  to  begin  Intelligence  Preparation 
of  the  Battlefield  (IPB). 


T-3  days 


Simulation  site  personnel  and  Blue  Forces  Controller  train  and  rehearse  interactors  and 
roleplayers. 

Roleplayers  and  training  audience  set  up  their  command  posts  in  the  exercise  area. 

Division  (EXCON)  roleplayers  rehearse  division  order  v«th  observers. 

Exercise  Director  and  COBRAS  Coordinator  conduct  final  readiness  check  of  exercise 
support. 

Brigade  Commander  and  XO  conduct  final  readiness  check  of  brigade  staff. 


Figure  B-4.  Planning  and  preparation  timeline  for  COBRAS  I  Brigade  Staff  Exercise 
implementation. 
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Option 


1: 


MTC-> 

AD-> 

DATK 


2:  MTC-» 
AD 


3:  MTC 


4:  AD-> 
DATK 


5:  AD 


6:  DATK 


Perwits  the  brigade’s  planning  process  to  begin  before  hostilities  and  continue  du'oughout 
die  three  missions,  and  requires  the  brigade  to  use  both  the  deliberate  decision-making 
process  (DDMP)  and  the  modified  decision-making  process.  This  option  provides  the  most 
robust  opportunity  to  practice  combat  service  support  (CSS)  tasks.  The  CSS  activity 
influences  the  operation  from  the  time  the  brigade  begins  its  transition  fi’om  the  field 
training  exercise  to  combat,  and  continues  through  the  completion  of  the  deliberate  attack 
(DATK). 

Starts  with  the  movement  to  contact  (MTC),  transitions  to  the  area  defense  (AD),  and  ends 
when  the  brigade  has  completed  its  defenses,  consolidated  its  forces,  and  knows  the  status 
of  its  units.  Both  the  DDMP  and  the  modified  decision-making  process  will  be  practiced, 
and  CSS  activities  are  ongoing.  The  brigade’s  posture  at  the  beginning  of  the  AD  depends 
on  its  losses  in  the  MTC  and  on  the  resupply  and  replenishment  that  take  place  during  the 
transition. 

Provides  a  logical  entry  into  the  exercise,  with  brigade  units  performing  reorgani2ation 
activities  and  reporting  their  status  as  the  staff  plans  the  mission  using  the  DDMP.  The 
exercise  ends  when  the  enemy  advance  guard  main  body  breaks  contact  and  assumes  a 
temporary  defense. 

The  AD  starts  with  the  brigade  still  in  limited  contact  with  the  enemy  following  the  MTC. 
The  brigade’s  readiness  posture  represents  what  a  brigade  could  expect  following  a  MTC. 
This  condition  will  require  the  brigade  to  rapidly  assess  its  combat  capability  and  conduct 
the  resupply,  replenishment,  and  other  CSS  activities  to  prepare  for  the  AD  mission.  The 
time  available  demands  that  the  brigade  use  the  modified  decision-making  process  to  plan 
the  AD. 

This  option  is  useful  when  the  brigade  does  not  wish  to  train  on  the  MTC.  It  is  a  difficult 
starting  point,  however.  The  participants  must  be  fully  read  into  the  scenario  and  be  ready 
to  take  control  at  a  point  when  the  brigade  is  very  active. 

The  initial  conditions  are  the  same  as  for  option  4,  with  the  brigade  just  completing  the 
MTC.  As  with  option  4,  the  entry  pomt  for  this  option  will  offer  a  considerable  challenge 
to  the  brigade  staff.  The  modified  decision-making  process  must  be  used  to  plan  the  AD. 
The  mission  continues,  terminating  as  the  brigade  consolidates  its  forces  as  in  option  2. 

The  brigade  has  (notionally)  completed  the  MTC  and  AD  missions  and  is  located  in  an 
assembly  area.  Its  readiness  posture  is  representative  of  a  unit  that  has  fought  the  previous 
two  missions.  The  CSS  activities  are  ongoing  as  the  brigade  continues  planning  (using  the 
DDMP)  and  preparations  for  the  attack  and  replenishes  and  improves  its  combat  readiness. 
The  exercise  terminates  as  the  brigade  seizes  its  objective  and  consolidates,  before 
replenishment  and  repair  which  would  precede  the  next  mission. _ _ 


Figure  B-5.  The  six  implementation  options  for  the  Brigade  Staff  Exercise. 

When  option  1  (from  Figure  B-5)  is  selected,  the  BSE  requires  14  8-hour  days.  The  other 
options  will  decrease  the  length  of  the  exercise,  but  will  also  decrease  the  extent  to  which  the  staff 
can  practice  their  decision-making  process. 

The  BSE  AARs  are  designed  to  be  conducted  by  the  Senior  Observer  throughout  the 
exercise,  with  an  AAR  for  each  segment  of  the  mission,  as  defined  in  Figure  B-6.  After  each 
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segment,  one  hour  is  set  aside  for  an  AAR  involving  the  brigade  commander  and  his  staff  as  the 
primary  audience.  The  AAR  discussions  focus  on  the  strengths  and  weaknesses  of  the  staff 
process.  During  the  AARs,  observers  guide  the  staff  to  recognize  their  weaknesses  and  direct 
them  toward  the  “discovery”  of  alternative,  more  correct  actions  as  outlined  by  the  DDMP,  the 
modified  decision-making  process,  and  the  COBRAS  tasks.  The  AAR  materials  help  establish  the 
link  between  staff  performance  in  the  just-completed  exercise  segment,  and  the  outcomes  of  the 
prior  segment(s). 


Movement  to  Contact  Segments 

"  Area  Defense  Segments  j  ^ 

Deliberate  Aftack  Segments 

Mission  Analysis 

Mission  Analysis 

COA  Development 

Modified  Decision-Making 
Process  Planning 

COA  Development 

Wargaming 

Wargaming 

COA  Comparison 

COA  Comparison 

Orders  Preparation 

Orders  Preparation 

Orders  Preparation 

Rehearsal 

Rehearsal 

Rehearsal 

Mission  Execution 

Mission  Execution 

Mission  Execution 

Consolidate/Reorganize 

Figure  B-6.  Exercise  segments  corresponding  to  scheduled  after  action  reviews. 

Every  aspect  of  the  exercise  conduct  phase,  as  well  as  the  preparation  phase,  is 
documented  in  the  BSE’s  TSP.  That  is,  the  TSP  contains  all  the  information  and  instructions 
needed  to  conduct  the  exercise.  Below,  the  BSE  TSP  is  described,  in  terms  of  its  organization 
and  contents. 

The  BSE  TSP 

The  basic  structure  or  orgarrization  of  the  BSE  TSP  was  designed  to  account  for  all  six 
implementation  options  described  in  Figure  B-5.  That  is,  there  is  not  a  separate  TSP  for  each 
option.  Rather,  the  TSP  consists  of  a  single  set  of  guides,  books,  and  simulation  system  tapes  that 
contain  both  instructions  and  materials  required  for  all  implementation  options.  Upon  the 
selection  of  any  implementation  option,  a  brigade  can  select,  assemble,  and  distribute  those 
materials  needed  for  that  option.  The  TSP  provides  the  guides  and  materials  for  each  training 
participant,  appropriate  for  his/her  role  in  the  exercise. 

A  detailed  description  of  the  contents  of  the  TSP  can  be  found  in  Giraves,  Campbell, 

Deter,  and  Quinkert  (1997).  A  broad  overview  of  the  organization  of  the  TSP,  however,  is 
provided  in  Figure  B-7,  and  a  brief  outline  of  the  contents,  is  presented  in  Figure  B-8. 
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Exercise 

Management 

•  Exercise  Guide  for  the  Exercise  Director,  COBRAS  Coordinator,  and  Blue 
Forces  Controller,  with  Appendixes 

Tactical  Materials 

•  Corps  Concept  (MTC,  AD,  and  DATK) 

•  Division  Order  emd  Tactical  Materials  (including  overlays) 

•  Scripted  and  hard-copy  messages 

Participant  Guides 
and  Materials 

Training  Audience: 

•  Training  Audience  Guide  (generic,  for  all  1 1  Primary  Training  Audience 
members) 

•  Initial  Situation  Packages  and  start  of  exercise  (STARTEX)  Position  Overlays 
(per  staff  member,  per  mission) 

•  Task  Lists  (per  staff  member,  per  mission) 

Observers: 

•  Observer  Guide  (generic,  for  all  6  Observers) 

•  Task  Lists  (per  Observer,  per  mission) 

•  Observer  AAR  Briefing  Materials 

Roleplayer  Teams: 

•  EXCON  Roleplayer  Guide  •  OPFOR  Controller  Guide 

•  Task  Force  1-5  Roleplayer  Guide  •  Cavalry  Troop  Roleplayer  Guide 

•  Task  Force  1-7  Roleplayer  Guide  •  Fire  Support  Roleplayer  Guide 

•  Task  Force  3-5  Roleplayer  Guide  •  Engineer  Battalion  Roleplayer 

•  Task  Force  1-80  Roleplayer  Guide  Guide 

.  Brigade  Support  Area  Roleplayer  *  ^r  Defeuae  Artillery  Roleplayer 

Guide  f^uide 

•  Initial  Situation  Packages  and  STARTEX  Position  Overlays  (per  roleplayer 
team,  per  mission) 

BBS  Interactors: 

•  Blue  Interactor  Guide 

•  Red  Interactor  Guide 

•  HICON/EXCON  Interactor  Guide 

Simulation 

Materials 

Tools  for  initializing  BBS  and  making  cheuiges  or  corrections: 

•  BBS  TOE  and  Initialization  Book 

•  BBS  Archive  Book 

•  BBS  System  Tapes 

Figure  B-7.  Organization  of  the  Brigade  Staff  Exercise  training  support  package. 


B-11 


Material  fdn4  \ 

■-•CV.-V'T: 

Exercise 

A  planning  and  preparation  timeline 

Guidance  on  controlling  the  training 

Managers 

Guidance  on  selecting  missions 

Mission-specific  exercise  information 

Guidance  on  long-term  planning  and 

Information  on  implementation  model 

preparation 

Guidance  on  near-term  preparation 

Division  orders 

Training 

Preview  of  exercise  and  its  intent 

Guidance  on  exercise  preparation 

Audience 

Information  on  audience  composition 

Road  to  war 

Performance  objectives 

Initial  situation  materials 

Description  of  observers  and  after 
action  reviews 

Task  lists 

Roleplayers 

Preview  of  exercise  and  its  intent 

EXCON: 

(Subordinate 

Guidance  on  exercise  preparation 

Information  on  distribution  of  tactical  materials 

and 

Guidance  on  exercise  conduct 

Scripted  messages 

Supporting 
Units,  Exercise 
Control 

Workstation  task  lists 

Response  guidelines 

Simulation  operating  procedures 

Information  on  division  and  higher  assets 

[EXCON], 

Information  on  communications 

Division  orders 

Opposing 

Road  to  war 

OPFOR: 

Force 

Subordinate  and  supporting  units: 

Exercise  and  role  overview 

[OPFOR]) 

Planning  guidance  and  job  aids 

OPFOR  story  line  and  training  plan 

Initial  situation  materials 

Mission-specific  scenario  descriptions 

OPFOR  Controller  tasks 

Observers 

Exercise  and  role  overview 

Senior  Observer  responsibilities 

Information  on  exercise  preparation 

Division  orders 

Information  on  exercise  conduct 

Task  lists 

Interactors 

COBRAS  rules  of  engagement 

Addendum  to  the  BBS  4.0  Warfighter’s  Guide 

(Blue  and  Red) 

Information  on  naming  units/icons 

(National  Simulation  Center  et  al.,  1994) 

Information  on  operational  states 

Force  description  materials 

Job  aid  for  reporting  combat  service 

Initialization  instructions 

support  information 

Information  on  Brigade/Battalion 

Battle  Simulation  (BBS)  menus 

Execution  materials  (Red  only) 

Simulation 

Modified  table  of  organization  and 

Information  on  assigning  units  to  workstations 

Site  Staff 

equipment 

Information  on  graphic  control  measures 

Information  on  initializing  the  Brigade 

Information  on  roleplayer  and  interactor  training 

Staff  Exercise  (BSE) 

Instruction  on  archive  tapes 

for  the  BSE 

Figure  B-8.  Description  of  the  Brigade  Staff  Exercise  training  support  package  contents. 
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The  Brigade  Staff  Vignettes 

The  COBRAS  I  brigade  staff  vignettes  are  short,  structured,  self-contained  training 
activities.  Each  vignette  focuses  on  a  specific  staff  process  event  and  on  specific  members  and 
groupings  of  the  brigade  staff  Each  vignette’s  activities  are  a  “snapshot”  of  a  segment  of  the 
entire  staff  process.  They  represent  extracts  of  activities  that  are  normally  performed  by  the  staff 
in  a  context-rich  situation.  That  is,  the  vignettes  lift  discrete  events  out  of  the  context  in  which 
they  are  normally  found  and,  for  training  purposes,  treat  them  in  isolation. 

Providing  “Thursday  morning”  practice  opportunities  is  the  key  to  the  vignettes’  value  to 
a  brigade.  Each  vignette  sets  up  an  environment  in  which  selected  members  of  the  brigade  staff 
can  focus  on  the  performance  of  the  activities  required  by  individual  segments  of  the  plan, 
prepare,  and  execute  processes.  Vignettes  are  well  suited  for  the  intangible  aspects  of  staff 
processes,  including  integration,  coordination,  synchronization,  and  the  establishment  of  roles  and 
associations.  They  require  relatively  little  time  to  prepare  for  and  execute  (e.g.,  one  to  two  days). 

This  section  describes  four  key  aspects  of  the  vignettes: 

•  the  structure  of  the  set  of  vignettes, 

•  the  training  objectives  and  tasks, 

•  the  implementation  model,  and 

•  the  structure  and  composition  of  the  TSP  that  includes  all  the  information  required  to 
implement  the  wgnettes. 

The  Structure  of  the  Set  of  Vignettes 

During  the  COBRAS  I  project,  developers  produced  thirteen  vignettes.  Two  of  the 
vignettes  were  simulation-based,  and  were  not  completed  during  the  COBRAS  I  effort.  This 
discussion  focuses  primarily  on  describing  the  1 1  live  simulation-based  vignettes. 

Vignette  scenarios  were  derived  from  the  scenario  storyline  that  underlies  the  BSE.  The 
wgnette  scenarios  differ  from  the  scenario  of  the  BSE,  however,  in  that  each  vignette’s  scenario 
was  modified,  as  necessary,  to  support  the  specific  training  objective  of  the  vignette.  That  is,  each 
vignette  scenario  sets  up  the  tactical  situation  that  requires  the  participation  of  a  selected  group  of 
the  brigade  staff  members  in  a  well-delineated  event. 

In  addition  to  scenario  design,  vignette  construction  also  required  determination  of 
training  participant  groups.  Although  the  training  audience  of  1 1  members  of  the  brigade  staff 
had  been  identified,  each  vignette  required  additional  definition  of  the  particular  small  group  that 
would  participate.  In  practice,  the  selection  and  design  of  scenario  events  and  the  determination 
of  training  audience  personnel  were  inseparably  linked.  Together,  the  two  design  parameters 
defined  the  scope  of  the  vignette;  that  is,  the  mission  “slice”  that  would  be  the  vignette  situation. 

Defining  the  vignette  scope  involved  an  initial  tentative  choice  of  event  and  participants, 
followed  by  gradual  refining  and  shaping  of  vignette  structure.  This  shaping  process  required 
developers  to  go  through  a  four-step  process  (with  multiple  iterations); 
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1.  Analyze  the  events  and  activities  before,  during,  and  after  the  selected  event. 

2.  Analyze  and  document  the  inputs  and  outputs  of  that  extended  event  context. 

3.  Document  all  of  the  participants  in  each  separable  segment  of  the  extended  event 
context. 

4.  Finally,  draw  “the  line”  between  activities  and  participants  that  will  be  in  the  vignette 
itself,  and  those  that  will  not  be  part  of  the  vignette. 

For  any  given  vignette,  the  products  of  these  four  steps  were  continually  modified  during 
development.  The  process  eventually  yielded  the  vignette  titles  and  designation  of  training 
audience  members  shown  in  Figure  B-9. 


' :  .Vigtietfe  Title. 

X^flirting  Audience 

1:  Plan  for  Dislocated  Civilians 

SI,  S2,  S4 

2;  Plan  Refuel  on  the  Move 

S4,  FSB  Commander 

3:  Develop  a  Concept  of  Service  Support 

S1,S4 

4:  Develop  Reconnaissance  and  Surveillance 

S2,  S3 

Plan 

5:  Conduct  Target  Development 

XO,  S2,  S3,  FSO 

6:  Develop  Air  Defense  Concept 

S2,  S3,  ADCOORD 

7:  Develop  Contingency  Plan 

S2,  S3,  FSO,  ENG 

8:  Conduct  Mission  Analysis 

XO,  SI,  S2,  S3,  S4,  FSO,  ENG,  ADCOORD 

9:  Develop  Courses  of  Action 

XO,  SI,  S2,  S3,  S4,  FSO,  ENG,  ADCOORD 

10;  Conduct  Course  of  Action  Analysis 

XO,  SI,  S2,  S3,  S4,  FSO,  ENG,  ADCOORD 

1 1 :  Conduct  Special  Staff  Rehearsal 

XO,  S2,  S3,  FSO,  ENG,  ADCOORD 

Figure  B-9.  Titles  and  training  participants  for  the  COBRAS  I  vignettes. 

The  events  of  the  COBRAS  I  vignettes  highlight  selected  aspects  of  the  staff  planning 
process.  These  1 1  events  are  based  on  the  requirements  outlined  in  the  DDMP  as  described  in 
FM  101-5  (DA,  1997).  They  are  executed  in  a  live  training  environment.  By  using  this  type  of 
simulated  environment,  resource  costs  are  kept  low,  and  the  training  becomes  more  accessible  for 
brigade  staff  development. 
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While  there  is  a  performance  logic  to  the  vignettes,  a  sequence  was  not  part  of  the  design. 
The  content  reflects  planning  requirements,  followed  by  the  steps  in  the  tactical  decision-making 
process.  This  is  the  sequence  in  which  they  are  listed  in  Figure  B-9.  But  the  vignettes  do  not 
build  on  each  other  or  rely  on  input  from  preceding  vignettes.  Each  vignette  can  be  conducted 
independently  of  other  vignettes. 

Vignette  Trainine  Objectives  and  Tasks 

The  focus  in  the  vignettes  is  on  the  performance  of  the  participant  group,  rather  than  on 
the  isolated  performance  of  any  individual  members.  Performance  descriptions,  which  eventually 
became  the  training  objectives  and  tasks,  were  produced  from  the  results  of  the  SPA  (described 
earlier).  That  is,  the  SPA  task  lists  were  examined  and  aggregated  across  persons  in  order  to 
produce  group  performance  descriptions. 

The  vignette  task  lists  serve  two  purposes:  They  give  the  training  participants  a  preview 
of  the  performance  expectations  for  the  vignette,  and  they  give  the  Training  Coordinator 
(described  below)  a  guide  for  observing  and  coaching  the  participants. 

The  Vignette  Implementation  Model 

As  with  the  BSE,  the  implementation  model  for  vignettes  is  described  in  terms  of 
participants  and  their  roles,  preparation,  and  conduct. 

Vignette  participants.  The  vignettes  are  designed  to  be  conducted  by  brigade  personnel 
and  within  brigade  resources.  A  designated  Training  Coordinator  is  responsible  for  preparing  and 
conducting  the  vignette.  The  other  participants  represent  the  training  audience  and  their 
supporting  staff. 

The  Training  Coordinator  position  is  usually  filled  by  the  brigade  XO.  The  XO  has  the 
overall  responsibility  for  training  and  overseeing  the  staff.  Additionally,  he  is  the  most  senior  and 
most  experienced  person  on  the  staff.  He  has  a  vested  interest  in  how  the  staff  conducts  itself, 
both  individually  and  in  coordination  with  each  other.  The  Training  Coordinator  is  responsible  for 
the  overall  accomplishment  of  the  vignette.  He/she  must  organize  all  of  the  participants  and  brief 
them  on  their  roles.  They  must  understand  all  aspects  of  the  vignette,  including  the  training 
emphasis,  tactical  background,  and  the  AAR  process.  The  specific  duties  of  the  Training 
Coordinator  include  the  following: 

•  scheduling  the  training, 

•  preparing  the  participants  for  training, 

•  observing  and  coaching  during  training,  and 

•  facilitating  the  AAR. 

The  other  participants  include  the  specific  vignette  training  audience  members,  as  shown  in 
Figure  B-9,  and  additional  personnel  from  staff  sections.  These  section  personnel  are  included  at 
the  discretion  of  the  XO  and  training  audience  members,  in  order  to  provide  realistic  assistance  to 
the  training  audience  and  to  broaden  the  training  value  provided. 

When  the  XO  is  a  member  of  the  training  audience,  he  is  generally  performing  as  an 
overseer,  coordinator,  and  synchronizer  of  the  staff  process.  He  also  functions  to  establish  and 
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enforce  staff  procedures.  These  functions  are  compatible  with  the  role  of  the  Training 
Coordinator  and  can  be  performed  concurrently. 

Vignette  preparation.  While  the  planning  for  the  conduct  of  a  vignette  is  accomplished 
by  the  Trmning  Coordinator,  there  are  preparation  activities  for  all  of  the  participants.  These 
activities  are  outlined  in  Figure  B-10. 


Planning  and  Preparation  Activities  for  the  Training  Coordinator  |  ^ 

Schedule  the  training 

Schedule  dates  and  times  for  an  administrative  brief,  participant  preparation, 
and  vignette  execution. 

Select  the  training  site 

The  vignettes  are  performed  in  command  post  (CP)  -like  areas.  A  classroom  is 
adequate;  no  separate  staff  work  areas  are  required. 

Notify  participants 

All  participants  should  be  notified  well  in  advance  of  the  time  and  location  of 
the  vignette  conduct. 

Conduct  self¬ 
preparation 

The  Training  Coordinator  needs  to  read  and  understand  the  training  support 
package  (TSP)  materials  for  the  vignette  to  be  conducted.  This  includes 
acquiring  a  solid  knowledge  of  the  tactical  and  after  action  review  materials. 

Conduct  administrative 
brief 

Once  the  vignette  has  been  scheduled,  the  Training  Coordinator  assembles  the 
participants  to  discuss  their  role,  distribute  participant  materials,  and  inform 
them  of  the  preparation  activities  they  must  complete  prior  to  vignette  execution. 

JPiniiiiiilis  nitd  ifor  the  i^iijdicncc  v 

Review  the  vignette 

Review  the  scope,  objective,  and  tasks  of  the  vignette  to  understand  the  limits  of 
the  requirements  and  how  the  vignette  activities  fit  into  the  entire  staff  process. 

Review  reference 
materials 

Each  training  participant  should  consult  the  list  of  references  included  in  the 

TSP  and  review  those  as  necessary.  The  lists  of  references  include  referrals  to 
current  doctrine,  techniques,  and  procedures  relevant  to  what  will  occur  in  the 
vignette. 

Read  the  tactical 
materials 

Participants  must  be  thoroughly  familiar  with  the  tactical  materials  before 
vignette  execution.  Each  training  audience  member  receives  tactical  materials 
commensurate  with  his/her  role  in  the  exercise. 

Copy  overlays  to 
acetate 

Paper  copies  of  overlays  must  be  transferred  to  acetate  prior  to  the  vignette’s 
conduct. 

Set  up  the  training  site 

Participants  should  establish  work  areas,  post  overlays,  and  obtain  any  CP 
supplies  and  accessories  that  are  normally  required  for  the  activities  to  be 
trained. 

Figure  B-10.  Description  of  vignette  planning  and  preparation  activities. 


Exercise  conduct.  The  exercise  conduct  phase  of  the  vignettes  requires  the  Training 
Coordinator  and  other  participants  to  be  familiar  with  the  tactical  preparation  materials.  The 
initial  event  of  the  conduct  phase  is  the  situation  brief,  conducted  by  the  Training  Coordinator.  In 
the  situation  brief,  participants  are  provided  any  additional  tactical  information  needed  to  conduct 
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the  vignette,  but  not  required  for  preparation  (e.g.,  commander’s  guidance).  Additionally, 
participants  may  ask  questions  during  the  brief  regarding  the  scope  of  the  vignette,  limitations  of 
the  training  environment,  or  performance  expectations. 

During  the  conduct  of  the  vignette,  the  Training  Coordinator  is  responsible  for  observing 
and  coaching  while  the  other  participants  perform  the  staff  activities  required  by  the  vignette 
situation. 

Following  execution  of  the  vignette,  the  Training  Coordinator  uses  a  prepared  list  of  AAR 
questions  to  facilitate  an  AAR.  The  Training  Coordinator  can  use  any  or  all  of  the  AAR 
questions,  together  with  his  own  observations,  to  focus  the  AAR  on  specific  performance  issues. 
The  accompanying  considerations  assist  him  to  further  focus  the  questions.  They  are  not 
“correct”  answers;  rather,  they  specify  the  critical  elements  of  each  performance  component.  The 
training  audience  is  not  passive  during  the  AAR,  but  should  be  proactive  in  directing  the  focus  of 
the  discussion  into  areas  they  identify  as  individual  and  staff  group  strengths  and  weaknesses.  It  is 
important  to  note  that  the  focus  of  the  exercise  and  the  AAR  is  on  staff  functioning  and  not  on  the 
tactics  of  the  brigade  operation. 

The  Vignette  TSPs 

Because  the  vignettes  were  to  be  independent,  stand-alone  exercises,  all  of  the  background 
and  instructions  was  to  be  provided  to  users  without  repeating  the  information  in  every  vignette- 
specific  component  of  the  TSP.  This  was  accomplished  by  gathering  all  of  the  general 
information  into  a  single  Guide  to  Use  and  Implementation  of  Vignettes,  which  serves  as  the 
training  management  component  for  the  vignettes. 

Individual  vignette  TSPs  contain  all  of  the  necessary  information  for  conducting  that 
vignette.  Each  vignette  TSP  consists  of  a  Training  Coordinator  Guide ^  which  also  contains 
tactical  materials  that  are  to  be  used  during  the  vignette,  and  a  Training  Participant  Guide, 
containing  the  tactical  materials  for  use  in  preparation. 

To  help  units  implement  the  vignettes,  individual  vignette  TSPs  have  similar  structure  and 
appearance.  This  was  accomplished  by  using  a  standard  vignette  shell,  to  which  vignette-specific 
information  would  be  added.  Of  course,  the  differences  among  vignettes  was  such  that  the  shell 
itself  served  more  as  an  outline  than  as  a  template.  The  TSPs  reflect  this  concept  more  in  their 
content  than  in  their  organization.  The  structure  and  a  general  description  of  the  contents  of  the 
vignette  guides  and  training  management  component  are  presented  in  Figure  B-1 1  and 
Figure  B-12. 
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Material  for 


Vignette  participants  (contained  in  General  orientation  to  the  vignettes,  as  well  as  vignette-specific 
the  vignette-specific  Training  information: 
farndpan,  Guide)  ,  List  of  tasks 

•  Description  of  the  scope  of  the  activity  involved  in  the  vignette 

•  References 

•  Preparation  guidance 

Tactical  materials  to  be  used  during  preparation  may  include  orders, 
overlays,  intelligence  summaries,  course  of  action  descriptions, 
annexes,  or  decision  matrixes,  as  appropriate  to  the  vignette. 

Distributed  to  all  primary  participants  (not  “other”  participants) 
during  the  Administrative  Brief,  several  days  before  the  vignette  takes 
place. 


Training  Coordinator  (contained  in  General  orientation,  as  well  as  specific  guidance  for  preparation  and 
the  vignette-specific  Training  conduct  of  the  vignette: 

Coordinator  Guide)  ^  Administrative  Brief  for  participants’  orientation 

•  Description  of  participant  preparation  activities  that  the  Training 
Coordinator  oversees 

•  Situation  Brief  to  begin  the  vignette;  may  provide  a  tactical 
update  or  implementing  instructions 

•  Suggested  processes  for  staff  performance  (job  aids  for 
participants) 

•  Models  of  expected  outcomes  or  products  (sample  products) 

•  Information  on  organization  and  conduct  of  after  action  reviews 

•  Staff  process  questions  to  facilitate  the  after  action  reviews 

Supplements  general  guidance  found  in  the  Guide  to  Use  and 
Implementation  of  Vignettes. 


Figure  B-1 1.  Description  of  contents  of  vignette  training  support  package  guides. 
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Guide  Overviev^TPiirpose’anH  Scppe'of  the  Guide  "" 

Introduction,  Format,  and 

Contents  of  the  Guide 

Tells  how  the  Guide  is  organized  and  who  to  contact  for  more 
information. 

F  requently  Asked  Questidrts  Ab( 

out  COBRAS  Vignettes  ^  ^  I 

About  Structured  Training 

COBRAS  program  description;  definition  of  vignettes,  structured 
training,  and  training  support  package  (TSP);  explains  how  vignettes 
are  used. 

About  Vignette  Content 

Vignette  activities,  participants,  and  the  training  mtent. 

About  Simulation 

Simulation-based  and  “live”  simulation  vignettes. 

About  Support  Requirements 

Support  requirements  for  simulation-based  and  “live”  simulation 
vignettes. 

About  Managing  Vignette 
Training 

Roles  and  responsibilities  of  the  brigade  and  the  Executive  Officer  in 
implementing  vignettes;  duties  of  the  Training  Coordinator. 

About  Other  Participants 

Who  else  may  or  should  participate. 

About  the  Brigade 

Commander’s  Role 

Brigade  commander  as  training  leader  and  as  participant. 

About  the  TSP 

Organization  and  content  of  the  TSP  materials;  how  to  obtain  more 
copies;  about  modifying  vignettes  for  unit  standard  operating  procedure 
and  unit  task  organization;  how  to  field  tactical  questions  from 
participants. 

About  Site  Requirements  and 
Supplies 

Supplies,  overlays,  maps,  and  physical  site  setup. 

About  Vignettes  and  Training 
Strategies 

Relationship  to  the  Brigade  Staff  Exercise  and  among  vignettes; 
incorporating  vignette  training  with  other  staff  training. 

About  Time  Requirements 

Actual  vignette  time  and  preparation  time,  preparation  activities. 

About  Training  Feedback 

Evaluation  and  feedback;  source  of  vignette  tasks  and  sample  solutions; 
how  to  prepare  for  and  conduct  the  after  action  review;  coaching. 

About  Implementing 

How  to  get  started. 

Figure  B-12.  Description  of  contents  of  vignette  training  management  component. 
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APPENDIX  C 


BASIC  METHODOLOGY  FOR  DEVELOPING  STRUCTURED  TRAINING 

Stmctured  training  is  a  term  applied  to  training  programs  that  are  deliberately  and 
purpose&lly  constructed  so  as  to  focus  on  specific  training  objectives  (C.  H.  Campbell,  Campbell, 
Sanders,  Flynn,  &  Myers,  1995;  C.  H.  Campbell  &  Deter,  1997;  C.  H.  Campbell,  Deter,  & 
Quinkert,  1997).  Development  requires  that  attention  be  paid  to  the  focus  of  the  training, 
standardization  of  training  events,  and  the  construction  of  a  training  support  package  (TSP)  that 
supports  the  focus  and  standardization.  Attention  to  each  of  these  aspects  is  incorporated  in  the 
methodology  described  in  the  above-referenced  Army  Research  Institute-published  guides.  There 
are  four  phases  to  the  development  methodology,  as  shown  in  Figure  C-1. 


Decisions  on; 

PHASE  1: 

Target  audience 

initial  Dedsions 

Training  context 

. .  "■ - C” 

Simulation  technology 

PHASE  2: 

PHASE  3; 

Designate  Training 

Design  Scenario  and 

Objedives 

r 

Exercise  Outline 

Identify  task  sources,  tasks,  &  standards 
Refine  task  list  for  simulation  support 
Select  tasks  that  support  the  mission 


Design  exerdse  scenario 

Prepare  exerdse  context  &  specifications 

Outline  events  and  build  exerdse 


PHASE  4: 

Develop  Training  Support 
Package 


7 


Design  TSP  structure 
Prepare  TSP  materials 


Fotmative  Evaluation:  life  cyde  process  of  examine  -  evaluate  -  refine 

Content  reviews.  Instructional  reviews,  map  and  roleplay  rehearsals,  pilot  tests  and  ® 
trial  implementations 


Figure  C-1.  The  methodology  for  development  of  structured  simulation-based  training. 
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In  Phase  1,  developers  specify  the  training  requirement  in  terms  of  the  content  (e.g., 
mission  and  enemy  type,  terrain,  time  constraints,  number  of  exercise  start  points,  difficulty  level), 
training  audience  (e.g.,  unit  type  or  echelon,  personnel  within  unit),  and  appropriate  training 
environment  (i.e.,  specific  simulator/simulation). 

Phase  2  and  Phase  3  are  usually  concurrent  activities,  with  a  great  deal  of  interaction 
between  specifying  the  training  objectives  for  the  stipulated  mission,  and  designing  the  scenario  to 
prompt  performance  of  the  training  objectives. 

Finally,  in  Phase  4,  developers  construct  and  try  out  all  of  the  written  and  simulator/ 
simulation-based  components  of  the  training  program,  including  materials  for  the  trainers  and  for 
the  participating  unit.  These  materials  are  referred  to  as  the  TSP,  which  is  critical  to  ensuring 
standardized  implementation. 

The  entire  design  and  development  process  is  supported  by  formative  evaluation  activities. 
These  include,  but  are  not  limited  to,  trial  implementations  of  the  emerging  program  and  TSP. 
Formative  evaluation  is  considered  to  be  a  continuous  product  improvement  process  that  extends 
throughout  the  life  of  the  development  effort. 

Methodology  for  Development  of  Structured  Small  Group  Exercises  (Vignettes) 

The  methodology  that  applies  to  development  of  vignettes  is  a  variation  of  the  basic 
methodology  described  above.  Complete  descriptions  of  the  activities  may  be  found  in 
C.  H.  Campbell,  Ford,  and  Campbell  (in  preparation).  As  with  the  basic  process,  there  are  four 
phases  in  the  development  of  vignettes,  as  shown  in  Figure  C-2. 

In  Phase  1,  developers  determine  the  vignette  user  (e.g.,  unit  type  or  echelon,  personnel 
within  unit),  vignette  context  (e.g.,  mission  and  enemy  type,  force  organization,  terrain,  time 
constraints),  appropriate  training  environment  (e.g.,  specific  simulator/simulation,  command  post, 
rehearsal  site),  and  other  constraints;  tentatively  identify  audience  and  event;  and  specify  the 
vignette  architecture. 

During  Phase  2,  the  limits  of  the  vignette  are  defined  in  terms  of  the  mission,  enemy, 
terrain,  troops,  time,  and  civilian  considerations  (METT-TC);  the  tactical  framework  for  the 
vignette  is  generated;  the  scope  of  the  vignette  event  is  specified;  and  developers  define  and  begin 
construction  of  the  simulation-based  or  scripted  materials. 

In  Phase  3,  the  critical  performance  requirements  and  standards  for  the  selected  audience 
and  event  are  defined.  These  are  eventually  translated  into  the  training  objectives,  task 
statements,  and  after  action  review  questions. 

During  Phase  4,  developers  construct  and  try  out  all  of  the  components  of  the  training 
program,  including  materials  for  the  trainers  and  for  the  participants  and,  as  appropriate,  the 
simulator/simulation-based  components. 
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PHASE  1: 

Prepare  Initial  Design 

Decisions  on  target  audience,  training 
context,  simulation  technology 

1 

PHASE  2: 

Design  Vignette  Scenario 

Description  of  the  tactical  event(s)  that 
provide  the  context  for  the  vignette 

- ^  :  ■  ■  •  i 

PHASE  3: 

Prepare  Detailed  Training 
Objectives 

Prepare  the  statements  of  behavioral 
objectives  and  develop  feedback  aids 

, - 

PHASE  4: 
Develop  Training  S 
Package  (TS 

Support 

P) 

Construct  and  evaluate  the  TSP 
components 

PROGRAM  EVACUATION;  Content  reylewSr  map/simulaliori 
exercises,  pitot  tests,  trianmpten^entations 


Figure  C-2.  The  four  phases  in  the  methodology  for  development  of  structured  vignettes. 


These  four  phases  are  not  done  in  a  strictly  linear  fashion.  There  will  be  considerable 
overlap  among  the  phases.  For  example,  understanding  of  the  initial  decisions  will  change  as 
more  is  learned  about  the  training  environment  and  the  performance  requirements;  information 
that  feeds  into  development  of  the  TSP  will  be  captured  during  documentation  of  initial  decisions; 
and  so  on. 

As  with  the  basic  methodology,  the  vignette  methodology  incorporates  a  series  of  required 
and  critical  formative  evaluation  steps  throughout  the  development  process.  Some  of  these  are 
formal  activities  (such  as  roleplay  enactments  of  the  vignette  conducted  by  the  design  team, 
tryouts  with  representative  individuals,  technology-driven  tryouts,  or  content  reviews  with  experts 
and  stakeholders),  but  formative  evaluation  also  includes  a  continuous  attention  to  the  need  for 
revisions  and  improvements.  The  purpose  of  formative  evaluation,  both  the  formal  activities  and 
the  continuous  revisions,  is  to  insure  the  quality  of  the  product  by  attending  to  quality  throughout 
development.  In  Figure  C-2,  formative  evaluation  is  shown  as  a  process  that  underlies  all  of  the 
design  and  development  work.  This  continuous  examine-evaluate-refine  process,  which  derives 
from  the  formative  evaluation  attention,  is  what  causes  design  and  development  to  be  non-linear. 
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